lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 10.2027 seconds; current memory usage: 0.177 Gibyte, delta: 0.040 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.4844 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.4986 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/pipeline_orig.qgraph DEEP/20220826/B1l/science#step0/20240307T150529Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 5.1839 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 17.4294 seconds; current memory usage: 0.318 Gibyte, delta: 0.141 Gibyte, peak delta: 0.128 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.318 Gibyte (main), 0.320 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0739 seconds; current memory usage: 0.318 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.318 Gibyte (main), 0.320 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 3720 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20220826_B1l_science#step0_20240307T150529Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 1.1875 seconds; current memory usage: 0.318 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.318 Gibyte (main), 0.320 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 3722 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 2.3529 seconds; current memory usage: 0.322 Gibyte, delta: 0.003 Gibyte, peak delta: 0.003 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.322 Gibyte (main), 0.320 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 14a3f74c-2371-4eea-962c-16569b09cc8d parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 0 executors parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22377792571472 parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-07 15:06:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-07 15:06:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 15:06:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-07 15:06:36 proc_lsst.multi:170 [INFO] [multi] job_id 53087 proc_lsst.multi INFO: [multi] job_id 53087 2024-03-07 15:06:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:36 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 53087 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 53087 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 53087 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 73a6cbf49f5ef46e847f42c4f79aee8b parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ba3b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/98bdeb89-2533-40dd-b017-91c8bd9198ee_isrForCrosstalkSources_1125906_45.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/98bdeb89-2533-40dd-b017-91c8bd9198ee_isrForCrosstalkSources_1125906_45.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 3668382aa555d99eaa1c5e4e8786d999 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ba3b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/8e6e6a84-b008-4cf0-922c-4a192f7a585e_isrForCrosstalkSources_1125940_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/8e6e6a84-b008-4cf0-922c-4a192f7a585e_isrForCrosstalkSources_1125940_27.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash b0cce9ed254d8b52811eae24411050bb parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ba3b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/dc2fddc1-ce59-410e-9811-87fce004b75b_isrForCrosstalkSources_1125937_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/dc2fddc1-ce59-410e-9811-87fce004b75b_isrForCrosstalkSources_1125937_57.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 1b3e44efc258b241091ff2c564306423 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ba3b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/4b53e8d4-07b0-4089-b25e-5f5b20cc3ba8_isrForCrosstalkSources_1125924_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/4b53e8d4-07b0-4089-b25e-5f5b20cc3ba8_isrForCrosstalkSources_1125924_49.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 356d301980b39f5470c23232c0b4f321 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ba3a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/db1cc29b-62d3-4e77-86e6-31a450abcc15_isrForCrosstalkSources_1125923_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/db1cc29b-62d3-4e77-86e6-31a450abcc15_isrForCrosstalkSources_1125923_56.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 848ee740662519e9ba15522b08e9176c parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ba3b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/6b34c6a4-a11e-4ffe-b729-2061134d1ffd_isrForCrosstalkSources_1125905_13.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/6b34c6a4-a11e-4ffe-b729-2061134d1ffd_isrForCrosstalkSources_1125905_13.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 92352105e1a9248e5e774eda1a6f0059 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ba3b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/150d1312-4cab-4e5e-9488-15f6f7400d19_isrForCrosstalkSources_1125906_37.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/150d1312-4cab-4e5e-9488-15f6f7400d19_isrForCrosstalkSources_1125906_37.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 2c3e6568de0ed0e3be1dd58eb1a99566 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ba3b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/2ad9bc49-1f7d-44f6-b7e8-37759a799852_isrForCrosstalkSources_1125919_48.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/2ad9bc49-1f7d-44f6-b7e8-37759a799852_isrForCrosstalkSources_1125919_48.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 4529941d2b71479badae19dd2459f9f8 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ba3b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/810670c6-f9ae-49a7-95b4-245f9a2bf2b8_isrForCrosstalkSources_1125943_36.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/810670c6-f9ae-49a7-95b4-245f9a2bf2b8_isrForCrosstalkSources_1125943_36.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 6efcba1d56ac6afb008ef242621296e3 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ba3bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/a92ef0be-cc6b-42a9-b761-d9ea513e4a9f_isrForCrosstalkSources_1125919_60.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/a92ef0be-cc6b-42a9-b761-d9ea513e4a9f_isrForCrosstalkSources_1125919_60.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash efe1a61d39a5b2f523c760a1394e6899 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ba3b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/ecaaf644-5219-4376-bf6f-d4c500186426_isrForCrosstalkSources_1125944_7.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/ecaaf644-5219-4376-bf6f-d4c500186426_isrForCrosstalkSources_1125944_7.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash a65444b890817ca6b8ae5ac7febf87b1 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ba3b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/c713b05f-0d83-4d4b-a5cb-89b74fdd358c_isrForCrosstalkSources_1125942_59.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/c713b05f-0d83-4d4b-a5cb-89b74fdd358c_isrForCrosstalkSources_1125942_59.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 3e7b6b36150b8a44f23c38b478d53f21 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ba3be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/34cfcaa4-0aec-489d-90bf-b19fb701bb22_isrForCrosstalkSources_1125907_3.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/34cfcaa4-0aec-489d-90bf-b19fb701bb22_isrForCrosstalkSources_1125907_3.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 430f82ed7633812ead111e1dff08aa49 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/5fe4ba29-d820-48ea-992d-0abc622a58b1_isrForCrosstalkSources_1125932_28.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/5fe4ba29-d820-48ea-992d-0abc622a58b1_isrForCrosstalkSources_1125932_28.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash ee5b06594ba2cb68b64f014320058be8 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/7cdcc8ea-6810-42c8-b0d3-0b0a591629ba_isrForCrosstalkSources_1125895_22.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/7cdcc8ea-6810-42c8-b0d3-0b0a591629ba_isrForCrosstalkSources_1125895_22.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 03479548d81a516c847ae4bf05a51380 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/93728636-a3ac-4b29-8671-0b935c0a86af_isrForCrosstalkSources_1125931_56.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/93728636-a3ac-4b29-8671-0b935c0a86af_isrForCrosstalkSources_1125931_56.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 365e35489285713e7878b7eed940caa5 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/30677ff4-043f-42db-a120-bb410320c937_isrForCrosstalkSources_1125936_18.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/30677ff4-043f-42db-a120-bb410320c937_isrForCrosstalkSources_1125936_18.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash d2976fabed2c6dc1a78e724c8f9193b4 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/d2a67096-128f-4b8b-a31d-606f8eb8038f_isrForCrosstalkSources_1125933_16.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/d2a67096-128f-4b8b-a31d-606f8eb8038f_isrForCrosstalkSources_1125933_16.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash b629429c3ccec3b1db866e355f3524dd parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/30fcc946-ad00-4835-a36d-d1ae86bd7e6c_isrForCrosstalkSources_1125902_27.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/30fcc946-ad00-4835-a36d-d1ae86bd7e6c_isrForCrosstalkSources_1125902_27.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 4094401696e3c968e0b5c195e1ef50d9 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/5682fc07-95bb-4709-9f83-fec63990a7c4_isrForCrosstalkSources_1125903_48.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/5682fc07-95bb-4709-9f83-fec63990a7c4_isrForCrosstalkSources_1125903_48.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 50014a4ef377ba9055011635267a7c21 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/9ca3a0fd-42f4-49c3-b6d3-97505f29b1e5_isrForCrosstalkSources_1125898_57.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/9ca3a0fd-42f4-49c3-b6d3-97505f29b1e5_isrForCrosstalkSources_1125898_57.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 1109c5f0852bfdf7ea4c097c55d8a464 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/015d8094-bf7d-4078-9674-9911c5415c67_isrForCrosstalkSources_1125919_62.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/015d8094-bf7d-4078-9674-9911c5415c67_isrForCrosstalkSources_1125919_62.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 41998944fbbac9c82d0a1551fc8dcde4 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f36a0baf-93ff-43ac-ac7b-18d6901c828b_isrForCrosstalkSources_1125897_48.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f36a0baf-93ff-43ac-ac7b-18d6901c828b_isrForCrosstalkSources_1125897_48.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 7090b80dd3788c5e73bdb583a05bacf9 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/efd30137-cc45-4c48-8afe-0cdf27209dae_isrForCrosstalkSources_1125903_19.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/efd30137-cc45-4c48-8afe-0cdf27209dae_isrForCrosstalkSources_1125903_19.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash bdef1aa1f00206cc6997d5ce20b4ceb7 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9c397d48-9c95-4026-bbe8-8064f1113ed3_isrForCrosstalkSources_1125921_11.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9c397d48-9c95-4026-bbe8-8064f1113ed3_isrForCrosstalkSources_1125921_11.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 70738b2f0c86c7090bcf37b803c42218 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/067690b7-318a-4088-b270-75459da3dbe9_isrForCrosstalkSources_1125923_55.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/067690b7-318a-4088-b270-75459da3dbe9_isrForCrosstalkSources_1125923_55.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 1ab99a83aeef6da1ad79994519635499 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0c3fe679-7751-490b-89c5-5e5fcffc77d2_isrForCrosstalkSources_1125890_24.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0c3fe679-7751-490b-89c5-5e5fcffc77d2_isrForCrosstalkSources_1125890_24.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash cc2c31c2fdf0ab0d8228fec073c1c7b6 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/25f86d2f-237b-4ba1-a704-777394b18320_isrForCrosstalkSources_1125894_14.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/25f86d2f-237b-4ba1-a704-777394b18320_isrForCrosstalkSources_1125894_14.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 7e92adce40120a569f114392243aa8ba parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/8506b516-2126-485e-a821-5b93d8f11d20_isrForCrosstalkSources_1125933_49.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/8506b516-2126-485e-a821-5b93d8f11d20_isrForCrosstalkSources_1125933_49.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 9fdd6878cc0bb6ba25892d8946318b8c parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/bdcf2eb2-ec37-4a83-88f4-8e4a98e56bf1_isrForCrosstalkSources_1125905_25.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/bdcf2eb2-ec37-4a83-88f4-8e4a98e56bf1_isrForCrosstalkSources_1125905_25.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 9951a4a1678b402056b0839c8c050016 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/f0b33342-0063-455c-a915-3b381f62f51f_isrForCrosstalkSources_1125927_18.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/f0b33342-0063-455c-a915-3b381f62f51f_isrForCrosstalkSources_1125927_18.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash ba7701fa01266c1e84f4caca359f51d5 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/ca5e3bde-9058-47e9-8ece-b411e79500c2_isrForCrosstalkSources_1125937_51.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/ca5e3bde-9058-47e9-8ece-b411e79500c2_isrForCrosstalkSources_1125937_51.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash b5fa84d2e60658cc9635dc31f48a0422 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/69fb2474-85a2-46a8-aa15-3a68a0b5eae3_isrForCrosstalkSources_1125890_31.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/69fb2474-85a2-46a8-aa15-3a68a0b5eae3_isrForCrosstalkSources_1125890_31.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 94e891b69ca653f3e2367e214b317e12 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f165198d-30b0-43cf-b992-2f7aec3cc189_isrForCrosstalkSources_1125922_18.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f165198d-30b0-43cf-b992-2f7aec3cc189_isrForCrosstalkSources_1125922_18.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 90d99311e6aac81c2d55eba3143a6c35 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/18ff2f1b-7803-4649-8041-da62e43fcdf2_isrForCrosstalkSources_1125946_32.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/18ff2f1b-7803-4649-8041-da62e43fcdf2_isrForCrosstalkSources_1125946_32.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash b5d581aedccc44f630b08a567fa84e26 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/4a571317-7ff9-480e-b8c1-0d67fe467fb9_isrForCrosstalkSources_1125892_60.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/4a571317-7ff9-480e-b8c1-0d67fe467fb9_isrForCrosstalkSources_1125892_60.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 53ca7f5c91a3b29132a49a72eeba15ae parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/c599ec5a-4a57-4daa-8707-30e623b4b2c2_isrForCrosstalkSources_1125936_10.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/c599ec5a-4a57-4daa-8707-30e623b4b2c2_isrForCrosstalkSources_1125936_10.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 495d1e9b1c86f23b6dd357cc80a6c21e parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/3e34db29-c1a9-4059-ba41-52f73b0d5d6d_isrForCrosstalkSources_1125931_10.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/3e34db29-c1a9-4059-ba41-52f73b0d5d6d_isrForCrosstalkSources_1125931_10.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 79646ae77039b8ead4df5c32b9fd57d8 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/ace97819-a2a7-4167-9284-2159de2654e5_isrForCrosstalkSources_1125940_29.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/ace97819-a2a7-4167-9284-2159de2654e5_isrForCrosstalkSources_1125940_29.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash b783485cf434f91fa89e4da8e01240a7 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/e89d2964-ad20-4784-abb9-821d847103ef_isrForCrosstalkSources_1125935_19.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/e89d2964-ad20-4784-abb9-821d847103ef_isrForCrosstalkSources_1125935_19.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash cde9f318e6f8fbf7e82af92ddf26808c parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/b7faa7cc-e2a2-458e-a82b-64e5bf64b3af_isrForCrosstalkSources_1125942_53.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/b7faa7cc-e2a2-458e-a82b-64e5bf64b3af_isrForCrosstalkSources_1125942_53.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash a59f96d69c03d285c8d0e51c77d42846 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/2a03a04f-3380-42f8-9145-31b6bb6ae884_isrForCrosstalkSources_1125934_1.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/2a03a04f-3380-42f8-9145-31b6bb6ae884_isrForCrosstalkSources_1125934_1.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 6135e3bf459376d6bd5d881658a32798 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/3bd8e76c-52c6-412f-9627-eff561f636c8_isrForCrosstalkSources_1125907_29.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/3bd8e76c-52c6-412f-9627-eff561f636c8_isrForCrosstalkSources_1125907_29.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 61479cfeafe35e266f86be5af3b068f1 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/9fade219-c30d-4c4a-ab2d-24e5da8f7dd7_isrForCrosstalkSources_1125902_34.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/9fade219-c30d-4c4a-ab2d-24e5da8f7dd7_isrForCrosstalkSources_1125902_34.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 908b0a7feafb27aad2736db134c36fbb parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/b4962d4b-7344-4b84-82a3-645692160406_isrForCrosstalkSources_1125931_36.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/b4962d4b-7344-4b84-82a3-645692160406_isrForCrosstalkSources_1125931_36.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 10aad420b8934d106c3f0a727f88629f parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/23a959a7-3b1e-4cdc-9eb9-cc25f98f196e_isrForCrosstalkSources_1125911_61.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/23a959a7-3b1e-4cdc-9eb9-cc25f98f196e_isrForCrosstalkSources_1125911_61.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash e5dd8dfc36a1f9ebd08e9554eb6daac7 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/0994fd91-c49c-4440-a3ba-53f052d1fe55_isrForCrosstalkSources_1125891_59.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/0994fd91-c49c-4440-a3ba-53f052d1fe55_isrForCrosstalkSources_1125891_59.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 5e03a5156bb0a87eb360b071e3bec1f4 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/b9aa431f-8f22-4de1-b0ba-1046641bb122_isrForCrosstalkSources_1125943_32.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/b9aa431f-8f22-4de1-b0ba-1046641bb122_isrForCrosstalkSources_1125943_32.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 91a7cd5e3980ee86fdaa2c87f45a8703 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8e71bdda-4b2a-410a-abc2-f9f75bff77e5_isrForCrosstalkSources_1125944_19.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8e71bdda-4b2a-410a-abc2-f9f75bff77e5_isrForCrosstalkSources_1125944_19.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 715a10e97632c8d1eb4debd30778705c parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/e518c075-ad88-45c4-9cb5-dde35bd25204_isrForCrosstalkSources_1125915_6.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/e518c075-ad88-45c4-9cb5-dde35bd25204_isrForCrosstalkSources_1125915_6.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 05aed38672b54a3b3395ff47e1c3fa0f parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/c4c6e8de-0e08-4ec6-bb0a-536c09ff69fe_isrForCrosstalkSources_1125926_53.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/c4c6e8de-0e08-4ec6-bb0a-536c09ff69fe_isrForCrosstalkSources_1125926_53.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 311af6d7c5dcb6a08eaf8c66c3080e8e parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/accae294-1402-44ec-a3bc-3f439bb098d7_isrForCrosstalkSources_1125920_2.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/accae294-1402-44ec-a3bc-3f439bb098d7_isrForCrosstalkSources_1125920_2.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 20b568341a1f3126cb104b6bb25d9417 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/61a54416-069f-4bab-81bc-1c732e5702f5_isrForCrosstalkSources_1125888_51.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/61a54416-069f-4bab-81bc-1c732e5702f5_isrForCrosstalkSources_1125888_51.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 438ca13030256bed0f3e0a689c375795 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/a9be5382-137e-46c1-b77a-6a1a20bf8a36_isrForCrosstalkSources_1125894_28.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/a9be5382-137e-46c1-b77a-6a1a20bf8a36_isrForCrosstalkSources_1125894_28.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 42f70afb086bfa8ee5c9cd9eb88865ee parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/c72de10f-cc60-4b34-99f5-3e82695f669a_isrForCrosstalkSources_1125900_56.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/c72de10f-cc60-4b34-99f5-3e82695f669a_isrForCrosstalkSources_1125900_56.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 2649b71ea495ccf823ed7bf0791da8b4 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/fbdc29a2-3afb-4354-ad20-7028f874a58e_isrForCrosstalkSources_1125908_51.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/fbdc29a2-3afb-4354-ad20-7028f874a58e_isrForCrosstalkSources_1125908_51.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 77821b0ad70d095b1ddadd710e0f8015 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/7bf2f716-7445-4db4-8777-0142862d92b0_isrForCrosstalkSources_1125892_16.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/7bf2f716-7445-4db4-8777-0142862d92b0_isrForCrosstalkSources_1125892_16.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 40dc44845f8ca21f69619ade179297ce parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/529eed37-d53f-4f39-9eba-45694d28a694_isrForCrosstalkSources_1125889_20.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/529eed37-d53f-4f39-9eba-45694d28a694_isrForCrosstalkSources_1125889_20.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 3216db4698eafc7165fc0968107b235f parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/4147bcdd-c20b-41c2-995f-b50aad399250_isrForCrosstalkSources_1125932_24.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/4147bcdd-c20b-41c2-995f-b50aad399250_isrForCrosstalkSources_1125932_24.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash ea454576ace87a71f29a3d9bdaabf8ae parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/f5b461e0-3b76-4ea2-9325-41cbf800cb25_isrForCrosstalkSources_1125915_32.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/f5b461e0-3b76-4ea2-9325-41cbf800cb25_isrForCrosstalkSources_1125915_32.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 053a31215cccbf27cb39ef59207bbebf parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9d093029-0178-466c-938b-6fb7008e4ced_isrForCrosstalkSources_1125921_20.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9d093029-0178-466c-938b-6fb7008e4ced_isrForCrosstalkSources_1125921_20.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash e6279e848ef956de1d47f91009457707 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/bbb2d810-414c-4911-a9d9-9ca37a2437bb_isrForCrosstalkSources_1125890_47.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/bbb2d810-414c-4911-a9d9-9ca37a2437bb_isrForCrosstalkSources_1125890_47.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 6adb746016bceea39082c0aaf1feb310 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/4871ec31-c4e0-4615-8853-307d9a9c0fba_isrForCrosstalkSources_1125921_5.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/4871ec31-c4e0-4615-8853-307d9a9c0fba_isrForCrosstalkSources_1125921_5.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash c65becfac8150f448cc9652c4e33e940 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/5df5e786-37af-455a-81c2-a898e65ef58e_isrForCrosstalkSources_1125891_13.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/5df5e786-37af-455a-81c2-a898e65ef58e_isrForCrosstalkSources_1125891_13.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash ff38e345b4f0fffbcf5662625ee9a4ec parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/778348fe-fdf7-4654-9985-72fe19bbadbc_isrForCrosstalkSources_1125923_14.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/778348fe-fdf7-4654-9985-72fe19bbadbc_isrForCrosstalkSources_1125923_14.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 9222c887a684ddd82ad6f03e38f24fb4 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/582d64b6-e2db-43f0-8171-436c45e920e6_isrForCrosstalkSources_1125891_49.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/582d64b6-e2db-43f0-8171-436c45e920e6_isrForCrosstalkSources_1125891_49.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash a3726f6879c6698d58aae4875e4226e0 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/8a95be13-6706-4a1d-96aa-23db49c7af77_isrForCrosstalkSources_1125908_7.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/8a95be13-6706-4a1d-96aa-23db49c7af77_isrForCrosstalkSources_1125908_7.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash d46b012fe13210f01eb12f30229509bc parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/abdb2c8d-8430-461d-a7ec-2fa8adf52f6e_isrForCrosstalkSources_1125943_12.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/abdb2c8d-8430-461d-a7ec-2fa8adf52f6e_isrForCrosstalkSources_1125943_12.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash b348017eaefb105116961ea88c5f8533 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/5195510f-4f78-455f-8505-9199bbdd7578_isrForCrosstalkSources_1125907_44.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/5195510f-4f78-455f-8505-9199bbdd7578_isrForCrosstalkSources_1125907_44.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash d896420e5ee1b063d14b8912397f4a99 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/951642b6-34f7-4252-a481-bbac6180cdcf_isrForCrosstalkSources_1125890_38.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/951642b6-34f7-4252-a481-bbac6180cdcf_isrForCrosstalkSources_1125890_38.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 5666b3a9d06e6a05f45ceced3cb4761b parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/0678c654-d0b0-42d7-b439-6505bc3089cb_isrForCrosstalkSources_1125913_39.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/0678c654-d0b0-42d7-b439-6505bc3089cb_isrForCrosstalkSources_1125913_39.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 38fc20839fafc2dc3c578d28c9268ac1 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/1a07b8cd-1856-490c-9dcc-6a48de027825_isrForCrosstalkSources_1125903_20.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/1a07b8cd-1856-490c-9dcc-6a48de027825_isrForCrosstalkSources_1125903_20.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash c7c0313a6fe3553f132a434dbe8218e2 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/5e1befb3-e89a-4eb1-9cac-696bf2fb6af3_isrForCrosstalkSources_1125897_31.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/5e1befb3-e89a-4eb1-9cac-696bf2fb6af3_isrForCrosstalkSources_1125897_31.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash dd462dfb7ef12347b0276442aa21fb4a parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f10d1891-5d74-470a-8fe1-d66dca0dc809_isrForCrosstalkSources_1125910_20.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f10d1891-5d74-470a-8fe1-d66dca0dc809_isrForCrosstalkSources_1125910_20.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 6dc54788bd46d3220e79e787c398aae3 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/42d132d2-bb00-42b0-a24b-2bf9433d6b07_isrForCrosstalkSources_1125915_28.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/42d132d2-bb00-42b0-a24b-2bf9433d6b07_isrForCrosstalkSources_1125915_28.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 6a4ee75a8b5e2a30fe145766d497d30f parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/e76dd1d9-2ef7-4375-855d-0afc442024b7_isrForCrosstalkSources_1125936_19.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/e76dd1d9-2ef7-4375-855d-0afc442024b7_isrForCrosstalkSources_1125936_19.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash c8061f6fdd5eaec561e8f759ba417e3b parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/46fa8c6f-ab64-4422-aca0-9383f4b53f77_isrForCrosstalkSources_1125903_62.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/46fa8c6f-ab64-4422-aca0-9383f4b53f77_isrForCrosstalkSources_1125903_62.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 65267526e1e2ba04c2bfc2db5eb9e8a4 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/9a79402f-d2ed-4b41-b7fb-c1ceebaea66a_isrForCrosstalkSources_1125889_48.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/9a79402f-d2ed-4b41-b7fb-c1ceebaea66a_isrForCrosstalkSources_1125889_48.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash c17fb74512168226f8e8b62288e9e621 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/aeb828c8-d8f3-44ab-a120-e6dffdf4db1a_isrForCrosstalkSources_1125894_24.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/aeb828c8-d8f3-44ab-a120-e6dffdf4db1a_isrForCrosstalkSources_1125894_24.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 61849744d915838a16122e3668d3d553 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/19e2f422-1aff-4747-b1a9-1615f84c065e_isrForCrosstalkSources_1125924_60.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/19e2f422-1aff-4747-b1a9-1615f84c065e_isrForCrosstalkSources_1125924_60.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 0d51ae8a8e28c521de735fdf04f1cfd2 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/8484f5dd-5d49-4e8e-b288-ccc53b214c10_isrForCrosstalkSources_1125894_34.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/8484f5dd-5d49-4e8e-b288-ccc53b214c10_isrForCrosstalkSources_1125894_34.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 0079216a0452e5f536cabd2fa94c8d61 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/2417ae47-05d9-40d6-bd34-679a1b06b9e9_isrForCrosstalkSources_1125891_48.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/2417ae47-05d9-40d6-bd34-679a1b06b9e9_isrForCrosstalkSources_1125891_48.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash ffe4d60877017046689f1b0bbef0598d parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/1880d524-f0a9-4bac-85d8-bbeeed80e33c_isrForCrosstalkSources_1125922_36.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/1880d524-f0a9-4bac-85d8-bbeeed80e33c_isrForCrosstalkSources_1125922_36.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash e728234a692c6b9c8b26be21e078f5fd parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/29714b9d-4e9a-4b73-a00e-ffc2c5656506_isrForCrosstalkSources_1125893_13.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/29714b9d-4e9a-4b73-a00e-ffc2c5656506_isrForCrosstalkSources_1125893_13.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash bc7e6208c9453bd9c3a764d5f7409dfd parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/0a2fd52d-f7c6-4f7f-ab98-daa3376f84ae_isrForCrosstalkSources_1125892_29.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/0a2fd52d-f7c6-4f7f-ab98-daa3376f84ae_isrForCrosstalkSources_1125892_29.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 4af414cf7c21b5881f0897c77219a0b0 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1a382e5e-caa5-43da-ab42-375b1932e1d4_isrForCrosstalkSources_1125900_54.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1a382e5e-caa5-43da-ab42-375b1932e1d4_isrForCrosstalkSources_1125900_54.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 2ae904777fc4dcb2e1471112902433e1 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/249acbbb-c9a3-4c8d-9ec4-b0fa96081c39_isrForCrosstalkSources_1125945_46.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/249acbbb-c9a3-4c8d-9ec4-b0fa96081c39_isrForCrosstalkSources_1125945_46.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash cce8905138973dd42a9a5ce00ee4d6a8 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/63b737cf-32b1-4025-ac1b-09f41b09bf68_isrForCrosstalkSources_1125928_50.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/63b737cf-32b1-4025-ac1b-09f41b09bf68_isrForCrosstalkSources_1125928_50.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 007c9ce3f8fc07c99efc86b75c077960 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/d3f290e4-89b0-4431-8141-b2031d291ef1_isrForCrosstalkSources_1125927_33.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/d3f290e4-89b0-4431-8141-b2031d291ef1_isrForCrosstalkSources_1125927_33.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 12c7a1713dddb0d04b61746affade09c parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/fbab929d-ef93-41ef-abd6-35ac760bd0f4_isrForCrosstalkSources_1125916_57.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/fbab929d-ef93-41ef-abd6-35ac760bd0f4_isrForCrosstalkSources_1125916_57.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 676424c298f95600c2f4a93f909e18cb parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/d3980308-2279-4232-9d95-7e2f3e322ef0_isrForCrosstalkSources_1125946_29.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/d3980308-2279-4232-9d95-7e2f3e322ef0_isrForCrosstalkSources_1125946_29.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 51e5585abd0c7018b96071f0418b57fa parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/9894abba-05b4-4c7c-a07e-0c20b56bbcb3_isrForCrosstalkSources_1125932_47.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/9894abba-05b4-4c7c-a07e-0c20b56bbcb3_isrForCrosstalkSources_1125932_47.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 4a2c632b3731fd47f9b319171146f230 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/e1e2ad77-600a-4307-9656-32062a97c84a_isrForCrosstalkSources_1125917_48.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/e1e2ad77-600a-4307-9656-32062a97c84a_isrForCrosstalkSources_1125917_48.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash d85e8f00848890da4c9ae2fa07f3942c parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/252ef4f6-b1f2-4445-ad21-575fe3dfde8a_isrForCrosstalkSources_1125942_35.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/252ef4f6-b1f2-4445-ad21-575fe3dfde8a_isrForCrosstalkSources_1125942_35.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash d366b10fec08af1f5dade7a879738fa1 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/8c3c7b2d-d8ff-4e24-89c4-cd9577e1cf7b_isrForCrosstalkSources_1125890_61.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/8c3c7b2d-d8ff-4e24-89c4-cd9577e1cf7b_isrForCrosstalkSources_1125890_61.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 2e0f743f003e40b91c9e175b58751979 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/736b7b58-60c4-48a1-bdfa-7691eac7db99_isrForCrosstalkSources_1125905_37.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/736b7b58-60c4-48a1-bdfa-7691eac7db99_isrForCrosstalkSources_1125905_37.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash a7fcacd79faa63724931817d292f1a71 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2af35943-fb81-484c-af44-4ed94e18fe54_isrForCrosstalkSources_1125895_35.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2af35943-fb81-484c-af44-4ed94e18fe54_isrForCrosstalkSources_1125895_35.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 85a39a90e1561089fd8e3a1a5aaf5e84 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d61392ee-1c21-4ffc-87f1-8b6bd20340ac_isrForCrosstalkSources_1125905_42.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d61392ee-1c21-4ffc-87f1-8b6bd20340ac_isrForCrosstalkSources_1125905_42.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash a598cf3676767c1685293b7dddb59b7a parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3e4b0288-36ee-41b6-89f6-194b529a3302_isrForCrosstalkSources_1125912_48.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3e4b0288-36ee-41b6-89f6-194b529a3302_isrForCrosstalkSources_1125912_48.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 4994d1432a49f51daf34f17f412b1b02 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f65cf5e4-9989-4013-ac78-cc14112b3fee_isrForCrosstalkSources_1125939_19.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f65cf5e4-9989-4013-ac78-cc14112b3fee_isrForCrosstalkSources_1125939_19.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 9a3435c03e0a873adb7e77c670145f2e parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/a2e5a877-1040-4edd-b5d8-bf6890bea9b3_isrForCrosstalkSources_1125926_57.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/a2e5a877-1040-4edd-b5d8-bf6890bea9b3_isrForCrosstalkSources_1125926_57.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 49fbe96e087d6001e4189cf86f866125 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/636a962d-402a-4025-8ed3-9603c6607f85_isrForCrosstalkSources_1125904_30.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/636a962d-402a-4025-8ed3-9603c6607f85_isrForCrosstalkSources_1125904_30.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash e04185c7efb356f6680f376ff153e998 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/3e931932-d454-45b2-8f05-9fb450b99325_isrForCrosstalkSources_1125896_35.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/3e931932-d454-45b2-8f05-9fb450b99325_isrForCrosstalkSources_1125896_35.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash a16774c79c412b9a1aba4f767f11e9a2 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/21cb01a3-d46c-4040-8126-d6cfe3693ec1_isrForCrosstalkSources_1125932_61.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/21cb01a3-d46c-4040-8126-d6cfe3693ec1_isrForCrosstalkSources_1125932_61.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 562c175975034294ad63694e116c9a18 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/7c1a4e12-8ae7-4ba4-b09f-7ccc01dea38c_isrForCrosstalkSources_1125944_28.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/7c1a4e12-8ae7-4ba4-b09f-7ccc01dea38c_isrForCrosstalkSources_1125944_28.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 3edf9d9c4ec794d67d5d55cdda12ff97 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/381c8b6e-4ea4-4660-80fa-494cc51aedbe_isrForCrosstalkSources_1125928_49.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/381c8b6e-4ea4-4660-80fa-494cc51aedbe_isrForCrosstalkSources_1125928_49.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 73f69cd58071d316c9cccbac22b6bc5f parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/897b9146-acf5-46b1-beab-f61822ddab22_isrForCrosstalkSources_1125888_8.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/897b9146-acf5-46b1-beab-f61822ddab22_isrForCrosstalkSources_1125888_8.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash c23d176a4dfaa956fca8936f13171d39 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/870ed48f-1f68-4f87-88c2-6d3ad77e2476_isrForCrosstalkSources_1125894_53.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/870ed48f-1f68-4f87-88c2-6d3ad77e2476_isrForCrosstalkSources_1125894_53.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 8c6e139424fd13ea5608a885e6141fb0 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/34a6cc6e-aaa4-4922-8227-637542c557b3_isrForCrosstalkSources_1125925_36.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/34a6cc6e-aaa4-4922-8227-637542c557b3_isrForCrosstalkSources_1125925_36.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 97f022d97f4ddbec711b8008b2573e1e parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/35ad8c02-1891-4f99-a4b8-c5d7040723fc_isrForCrosstalkSources_1125944_61.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/35ad8c02-1891-4f99-a4b8-c5d7040723fc_isrForCrosstalkSources_1125944_61.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 24fbbc8eb50f22ec9f55e325c6a69179 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/732a12a3-8689-41b8-9394-c7793c90f2f6_isrForCrosstalkSources_1125891_29.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/732a12a3-8689-41b8-9394-c7793c90f2f6_isrForCrosstalkSources_1125891_29.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash cce2699bebec9639b36124b40b5c00ab parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/cd72e14d-3b21-42e5-8dd2-d4a7bf0c497d_isrForCrosstalkSources_1125917_3.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/cd72e14d-3b21-42e5-8dd2-d4a7bf0c497d_isrForCrosstalkSources_1125917_3.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash de239ce5170d73885830fb0cb36846e5 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/13852863-445f-421a-a151-dfbf3c8e7004_isrForCrosstalkSources_1125944_38.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/13852863-445f-421a-a151-dfbf3c8e7004_isrForCrosstalkSources_1125944_38.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash c2aa59227d72d0b643de2b1aeb068871 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c3a3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/37420223-c466-42bd-85a9-7e4fcf048ae8_isrForCrosstalkSources_1125908_37.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/37420223-c466-42bd-85a9-7e4fcf048ae8_isrForCrosstalkSources_1125908_37.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash be31080a12f1fa8129805ecc6ed5ee9b parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd040e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/34d27b8d-31a6-4370-b138-8119fa0d6551_isrForCrosstalkSources_1125902_59.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/34d27b8d-31a6-4370-b138-8119fa0d6551_isrForCrosstalkSources_1125902_59.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 9d9f6ac451ac86010f5d111e6ebb77d1 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/d1f52508-f22e-49fa-8ac3-217af1ea50db_isrForCrosstalkSources_1125907_42.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/d1f52508-f22e-49fa-8ac3-217af1ea50db_isrForCrosstalkSources_1125907_42.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash aa78a33f81601d0ad3c2400a87c40b9e parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/1e75d3be-021d-4e40-b512-77105f6b6edc_isrForCrosstalkSources_1125929_7.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/1e75d3be-021d-4e40-b512-77105f6b6edc_isrForCrosstalkSources_1125929_7.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 7e3d36a28da62575fc7b71e4fd2de9fb parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/efa687d1-59b9-4aa4-a62f-4212bbe46b89_isrForCrosstalkSources_1125945_4.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/efa687d1-59b9-4aa4-a62f-4212bbe46b89_isrForCrosstalkSources_1125945_4.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash ba30123a0e21ce56c37ce34801fd3719 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd042c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/67a3ed79-c8c7-4a34-a77b-c0f1e1086f2a_isrForCrosstalkSources_1125911_7.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/67a3ed79-c8c7-4a34-a77b-c0f1e1086f2a_isrForCrosstalkSources_1125911_7.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 08f79a6a0caa97043a9451a8cad05c51 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a589e666-ef6a-4f13-ba7b-070b566b2825_isrForCrosstalkSources_1125938_25.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a589e666-ef6a-4f13-ba7b-070b566b2825_isrForCrosstalkSources_1125938_25.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash c51bf804b0953dc87f6c096005d8aa60 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd044a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/68fe0192-4eda-4249-bbe9-f5ef133d6e86_isrForCrosstalkSources_1125903_25.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/68fe0192-4eda-4249-bbe9-f5ef133d6e86_isrForCrosstalkSources_1125903_25.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash b2197304b6c34b1f7265cca6567cf8c2 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/982e2482-dcab-4d3b-8f72-38ce4ee416ef_isrForCrosstalkSources_1125919_29.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/982e2482-dcab-4d3b-8f72-38ce4ee416ef_isrForCrosstalkSources_1125919_29.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 803a6d91cc6e277fbd75e12f3001b338 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/74d27729-d5a2-449b-9fbc-8ffd00ebbbf9_isrForCrosstalkSources_1125916_26.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/74d27729-d5a2-449b-9fbc-8ffd00ebbbf9_isrForCrosstalkSources_1125916_26.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 72229eb25c5a65a16b40aefae96ad172 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd045e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/10fc6be7-9d63-4d5a-b5c2-d5a3170e761b_isrForCrosstalkSources_1125896_44.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/10fc6be7-9d63-4d5a-b5c2-d5a3170e761b_isrForCrosstalkSources_1125896_44.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 0a55fb83a570a2c266e5da52ea08ffb6 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/99a057c7-ea39-464d-acc3-7fd8ace0ee62_isrForCrosstalkSources_1125924_6.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/99a057c7-ea39-464d-acc3-7fd8ace0ee62_isrForCrosstalkSources_1125924_6.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 3444a2f1d0d4c9ddf2e33ba9f9a5e6de parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd047c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/1d520af2-6392-4026-a1db-5188a396ffba_isrForCrosstalkSources_1125898_32.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/1d520af2-6392-4026-a1db-5188a396ffba_isrForCrosstalkSources_1125898_32.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 10b9d2f2e029b0c46aca79cf9e3d287e parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/2fba4068-59f9-42ea-bf7f-f583d6490996_isrForCrosstalkSources_1125893_16.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/2fba4068-59f9-42ea-bf7f-f583d6490996_isrForCrosstalkSources_1125893_16.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash b8954f1ba26dde7c9a1975481826d331 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd049a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/a42b73ef-2779-4d2a-9043-acf31f91777b_isrForCrosstalkSources_1125902_17.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/a42b73ef-2779-4d2a-9043-acf31f91777b_isrForCrosstalkSources_1125902_17.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 349ead40ddb1ad0cd4770d91d8e9c104 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/b3c27f80-46ab-4c1b-92d2-d15d4cc23ff5_isrForCrosstalkSources_1125903_36.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/b3c27f80-46ab-4c1b-92d2-d15d4cc23ff5_isrForCrosstalkSources_1125903_36.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash ee2df4278b25db4f7b6bee54e1de9da4 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/c0c14190-d1f9-4dbb-9473-dba12b43f2f2_isrForCrosstalkSources_1125933_23.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/c0c14190-d1f9-4dbb-9473-dba12b43f2f2_isrForCrosstalkSources_1125933_23.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 07a298345eaed794e1a3014a42ed9e9a parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/7cca0f2e-7898-46ac-94e5-0b81b8a6eb7d_isrForCrosstalkSources_1125898_58.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/7cca0f2e-7898-46ac-94e5-0b81b8a6eb7d_isrForCrosstalkSources_1125898_58.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash b70d92d14fa42efdc81be920fea36e59 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/27d815e8-e864-4b3d-a915-8f927d9d8709_isrForCrosstalkSources_1125929_19.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/27d815e8-e864-4b3d-a915-8f927d9d8709_isrForCrosstalkSources_1125929_19.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 24531f05ccc6918c9787cc0c3306cd38 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/c48044fa-0ed4-4888-848d-6306c4595c8d_isrForCrosstalkSources_1125940_52.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/c48044fa-0ed4-4888-848d-6306c4595c8d_isrForCrosstalkSources_1125940_52.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash cddbfff789d4839d5131bac0869e8634 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/14177429-dd95-48cd-9e78-af1ba5d9da8d_isrForCrosstalkSources_1125888_53.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/14177429-dd95-48cd-9e78-af1ba5d9da8d_isrForCrosstalkSources_1125888_53.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 4cd96ce6fcbc26010181d96b16581fbd parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/45ab2447-cfef-488a-946a-5a0920843ca4_isrForCrosstalkSources_1125916_31.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/45ab2447-cfef-488a-946a-5a0920843ca4_isrForCrosstalkSources_1125916_31.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 67fcb8b289c3d802a833a6f69adab0e9 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/f70fbca4-fe7b-4d1a-a7dc-fc2fcbc7f7fd_isrForCrosstalkSources_1125932_20.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/f70fbca4-fe7b-4d1a-a7dc-fc2fcbc7f7fd_isrForCrosstalkSources_1125932_20.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash a36b6681fae366f7b1054cd1e0853d7a parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/c70831fd-bc2b-4646-b39f-b0dfcefb2eaf_isrForCrosstalkSources_1125940_10.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/c70831fd-bc2b-4646-b39f-b0dfcefb2eaf_isrForCrosstalkSources_1125940_10.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 29dbf18391153424232e6eec5f41eb1a parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/15a631c7-1465-448e-82e3-89524dc7dca1_isrForCrosstalkSources_1125892_58.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/15a631c7-1465-448e-82e3-89524dc7dca1_isrForCrosstalkSources_1125892_58.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash c7db7eebf58c6781e8aeef58aab3462d parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/b5a32698-416f-4680-8ee9-d0bf0679ccea_isrForCrosstalkSources_1125913_15.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/b5a32698-416f-4680-8ee9-d0bf0679ccea_isrForCrosstalkSources_1125913_15.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash eb562bd80810ca44ccd85aca317978c6 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/1583bcf5-0154-43a8-8dc1-0cdd2e89c3d1_isrForCrosstalkSources_1125909_55.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/1583bcf5-0154-43a8-8dc1-0cdd2e89c3d1_isrForCrosstalkSources_1125909_55.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash ecbcd1bc1c1905a366d8ad3f9de800a2 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9fe71f8f-5b6d-41e6-bdd0-8a0bf91671bd_isrForCrosstalkSources_1125921_3.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9fe71f8f-5b6d-41e6-bdd0-8a0bf91671bd_isrForCrosstalkSources_1125921_3.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash a003a0d0941997cf501d29b4bedc795d parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd051c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ce6a4b7a-b1d0-4f2c-a75e-8c59302d2786_isrForCrosstalkSources_1125894_27.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ce6a4b7a-b1d0-4f2c-a75e-8c59302d2786_isrForCrosstalkSources_1125894_27.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 00091618b6ff5df83a4def6d8d7026b7 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/2169826b-3702-4aca-8ab8-faee5e21aac4_isrForCrosstalkSources_1125914_35.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/2169826b-3702-4aca-8ab8-faee5e21aac4_isrForCrosstalkSources_1125914_35.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 0c2fbc2f6881e76a9191b4944c12e950 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/352ef2fe-fca3-4285-aa2b-29ef21210ce6_isrForCrosstalkSources_1125931_59.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/352ef2fe-fca3-4285-aa2b-29ef21210ce6_isrForCrosstalkSources_1125931_59.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash e4fa86c45a78a7d208fda23242dac255 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd053a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f67891c5-343d-4156-a7e2-97b10272d62f_isrForCrosstalkSources_1125930_18.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f67891c5-343d-4156-a7e2-97b10272d62f_isrForCrosstalkSources_1125930_18.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 06aacfd39c6e7b764b980c86f4dc560d parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/6cb3a710-c700-436c-bcda-03fdb3a825a2_isrForCrosstalkSources_1125914_62.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/6cb3a710-c700-436c-bcda-03fdb3a825a2_isrForCrosstalkSources_1125914_62.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 5a20e22245747195190014469140dc5c parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd054e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/12eaf205-8d78-430a-b089-b9683afe48c9_isrForCrosstalkSources_1125929_23.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/12eaf205-8d78-430a-b089-b9683afe48c9_isrForCrosstalkSources_1125929_23.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 723990e198c7b64e0f84103999f98034 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/02fc6273-18c7-4a2b-bd99-81df2df7cfca_isrForCrosstalkSources_1125891_9.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/02fc6273-18c7-4a2b-bd99-81df2df7cfca_isrForCrosstalkSources_1125891_9.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 8e4299f0adea0517df3ef060b495fd36 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/8b6fa6f6-f9a7-494e-8970-4fd0ab23f692_isrForCrosstalkSources_1125930_44.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/8b6fa6f6-f9a7-494e-8970-4fd0ab23f692_isrForCrosstalkSources_1125930_44.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 0c0022f8ceb2ba93256c496998b5e129 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd056c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a00fee5b-f6bd-4e1b-b558-be184e33f027_isrForCrosstalkSources_1125933_13.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a00fee5b-f6bd-4e1b-b558-be184e33f027_isrForCrosstalkSources_1125933_13.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash c41c04c9342949fc31a620e92b350097 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/b3c82e66-8cf8-4f89-a625-f6a47b1304b5_isrForCrosstalkSources_1125910_32.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/b3c82e66-8cf8-4f89-a625-f6a47b1304b5_isrForCrosstalkSources_1125910_32.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 7e7a8675eef84b1193966d924d3d49e7 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/6bfda748-565c-47d1-ab8e-311a7db32289_isrForCrosstalkSources_1125943_44.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/6bfda748-565c-47d1-ab8e-311a7db32289_isrForCrosstalkSources_1125943_44.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash dcb4537f37e882ad7e5f2d12673b0531 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/1cd98e05-1f80-4412-a39b-fef696d6009b_isrForCrosstalkSources_1125916_8.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/1cd98e05-1f80-4412-a39b-fef696d6009b_isrForCrosstalkSources_1125916_8.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 7e673574017ba00184ed6a3de916182c parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd058a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/3ec75d67-7c51-4b09-9610-850f41f368db_isrForCrosstalkSources_1125913_53.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/3ec75d67-7c51-4b09-9610-850f41f368db_isrForCrosstalkSources_1125913_53.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 44dea3ab1b3fca0c292e0ef33bcaef2b parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd059e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/6870bee0-c350-4c9c-8233-816a24383c3a_isrForCrosstalkSources_1125897_21.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/6870bee0-c350-4c9c-8233-816a24383c3a_isrForCrosstalkSources_1125897_21.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash a6b65e1df6f9ab66b3603435c587bbe8 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/d23de85d-acde-49c3-a701-8ffb2635b2e9_isrForCrosstalkSources_1125904_4.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/d23de85d-acde-49c3-a701-8ffb2635b2e9_isrForCrosstalkSources_1125904_4.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 3c4320c03db05174db318343dbe8b6a3 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/1e808b20-9309-4d2c-8692-cb9ed1e937d1_isrForCrosstalkSources_1125931_28.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/1e808b20-9309-4d2c-8692-cb9ed1e937d1_isrForCrosstalkSources_1125931_28.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 7c3d5428c0c0848658f0c42eedc59f8d parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/124634f3-e45c-41d9-a6c5-8b19dc37d26d_isrForCrosstalkSources_1125922_60.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/124634f3-e45c-41d9-a6c5-8b19dc37d26d_isrForCrosstalkSources_1125922_60.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash e3dc60a003d597f44b28b41ce20d441f parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b207b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/0b1ab236-65c2-4eff-82c6-9d82aaf42945_isrForCrosstalkSources_1125901_21.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/0b1ab236-65c2-4eff-82c6-9d82aaf42945_isrForCrosstalkSources_1125901_21.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash df923e0573b85cd0b03e3059a1886454 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a4db977e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/5ca828bc-9f09-481b-acfb-b8c3cc97c050_isrForCrosstalkSources_1125889_6.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/5ca828bc-9f09-481b-acfb-b8c3cc97c050_isrForCrosstalkSources_1125889_6.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash c6e4cc18589317350136f91cdc754783 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/40e989d0-99fe-4630-8403-9c15f4fdefb3_isrForCrosstalkSources_1125907_52.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/40e989d0-99fe-4630-8403-9c15f4fdefb3_isrForCrosstalkSources_1125907_52.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 097d823d3d18de31729cbd8f45cf0415 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd04040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/3bf68d9a-906b-48c1-bd5a-e9fa5760fb2b_isrForCrosstalkSources_1125904_55.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/3bf68d9a-906b-48c1-bd5a-e9fa5760fb2b_isrForCrosstalkSources_1125904_55.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash c40794c2025df8cf7d0805507fa669df parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/875dc908-4886-49cc-8c11-f65a0ce412c4_isrForCrosstalkSources_1125943_45.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/875dc908-4886-49cc-8c11-f65a0ce412c4_isrForCrosstalkSources_1125943_45.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 28e9a0ae20b55c2a01cc49826363f6fd parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/d368dd3a-b708-44da-92c6-9152bba4da44_isrForCrosstalkSources_1125921_15.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/d368dd3a-b708-44da-92c6-9152bba4da44_isrForCrosstalkSources_1125921_15.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 47606efb6e1eace9713535930c6377fb parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/e08ceb83-43d3-4616-b4ea-29cfd398b063_isrForCrosstalkSources_1125899_7.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/e08ceb83-43d3-4616-b4ea-29cfd398b063_isrForCrosstalkSources_1125899_7.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 769c7d6a49d43c49c8d2b6e6ba99d715 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/39d8228e-c0b9-4b03-b8bc-9662cf7f4930_isrForCrosstalkSources_1125914_46.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/39d8228e-c0b9-4b03-b8bc-9662cf7f4930_isrForCrosstalkSources_1125914_46.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 7d5c65a56eec662bac22b231f67a2750 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/d866a3e5-3e5f-41a4-b297-a6d254cb1401_isrForCrosstalkSources_1125932_5.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/d866a3e5-3e5f-41a4-b297-a6d254cb1401_isrForCrosstalkSources_1125932_5.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 2ad455f41a9d5f4b1ef64d5b5fc25486 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/6b81133c-8466-42fa-9a32-4b29b050696e_isrForCrosstalkSources_1125933_42.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/6b81133c-8466-42fa-9a32-4b29b050696e_isrForCrosstalkSources_1125933_42.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash a148013c205d6c2281dfd60b906d7887 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd060c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/8b6c58e7-aa70-427b-9ac2-5b411faf0d5e_isrForCrosstalkSources_1125924_54.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/8b6c58e7-aa70-427b-9ac2-5b411faf0d5e_isrForCrosstalkSources_1125924_54.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 4148fc4f441c0a7ca705094a1fb431d6 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/fd818e34-93e3-471f-9131-414591e28826_isrForCrosstalkSources_1125942_31.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/fd818e34-93e3-471f-9131-414591e28826_isrForCrosstalkSources_1125942_31.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash f8c96e2eb933f11cb0718d50c280df80 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd062a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/fbcde238-809a-45a0-8621-5a4ac58c541f_isrForCrosstalkSources_1125933_56.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/fbcde238-809a-45a0-8621-5a4ac58c541f_isrForCrosstalkSources_1125933_56.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash cda6290eef5f90f450be78ab2c0941fa parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/5a5aa490-fda3-407d-a3cb-d711e6ccfd60_isrForCrosstalkSources_1125913_61.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/5a5aa490-fda3-407d-a3cb-d711e6ccfd60_isrForCrosstalkSources_1125913_61.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 9a05788ece13226fd5a9b7bb000b98ca parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/9a92e92c-c071-48ef-940d-b01f81fb19c1_isrForCrosstalkSources_1125904_41.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/9a92e92c-c071-48ef-940d-b01f81fb19c1_isrForCrosstalkSources_1125904_41.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 21abda7800122fd7c02fd6692fe2b988 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd063e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/0d1026d4-baa3-47dd-aea6-5ca7ad490272_isrForCrosstalkSources_1125905_26.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/0d1026d4-baa3-47dd-aea6-5ca7ad490272_isrForCrosstalkSources_1125905_26.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 4223c0ad26244da6f022482c90f9da9c parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d0c61ff1-1178-4933-b89a-fa78eb187790_isrForCrosstalkSources_1125890_5.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d0c61ff1-1178-4933-b89a-fa78eb187790_isrForCrosstalkSources_1125890_5.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 288ad8ef1d551e08cac0904e565f2324 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd065c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/1dc8ab4b-67ec-467a-bf05-701dbbd60d8a_isrForCrosstalkSources_1125919_8.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/1dc8ab4b-67ec-467a-bf05-701dbbd60d8a_isrForCrosstalkSources_1125919_8.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 18ab91bc9ba24e7940eb144a4d29d506 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/104abc40-de73-42e1-b0a3-dac873cf2a57_isrForCrosstalkSources_1125912_3.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/104abc40-de73-42e1-b0a3-dac873cf2a57_isrForCrosstalkSources_1125912_3.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash f99b89fe1e9ffe77d2a20b1c9abd8d74 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd067a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/32e622d1-8143-44d2-a017-337194921ac9_isrForCrosstalkSources_1125894_20.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/32e622d1-8143-44d2-a017-337194921ac9_isrForCrosstalkSources_1125894_20.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 38e34be71947fc8a7ed948e905b55d8a parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/2f36088b-2ec8-49a7-bb19-b0cb577b9427_isrForCrosstalkSources_1125929_50.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/2f36088b-2ec8-49a7-bb19-b0cb577b9427_isrForCrosstalkSources_1125929_50.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 7ccf345eedd7b072d3c369d9557e7fa6 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ce9121f1-c70a-4baa-b618-e4ef46cce3ee_isrForCrosstalkSources_1125947_23.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ce9121f1-c70a-4baa-b618-e4ef46cce3ee_isrForCrosstalkSources_1125947_23.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 915430bbfc431ef08b2d43579623c142 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd068e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/f02454e9-5731-4b7a-b55e-f14a7e6b0568_isrForCrosstalkSources_1125938_58.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/f02454e9-5731-4b7a-b55e-f14a7e6b0568_isrForCrosstalkSources_1125938_58.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 43057276427e5a3da133100b58a41dab parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/94552628-df7b-473d-9d6c-48934f87876c_isrForCrosstalkSources_1125908_61.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/94552628-df7b-473d-9d6c-48934f87876c_isrForCrosstalkSources_1125908_61.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash aa9b44a7df12b6135ef81b0f9c17671c parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/e32e9e8b-e0d6-48cd-a646-968e6a6f9ad9_isrForCrosstalkSources_1125919_36.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/e32e9e8b-e0d6-48cd-a646-968e6a6f9ad9_isrForCrosstalkSources_1125919_36.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 5984b698cef65dce95a59a2b283d0f72 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/125d83bb-feb4-400b-ad61-3ba5f27aa121_isrForCrosstalkSources_1125908_49.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/125d83bb-feb4-400b-ad61-3ba5f27aa121_isrForCrosstalkSources_1125908_49.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash ea4938510a31bc854adff79b9ca3e5c5 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/a26318d6-e318-472f-950a-85d1c75067df_isrForCrosstalkSources_1125944_30.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/a26318d6-e318-472f-950a-85d1c75067df_isrForCrosstalkSources_1125944_30.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 50070c9f97659c0c33872bf52cf1032c parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/d002acac-1111-4e9a-83fd-70109ad90c7c_isrForCrosstalkSources_1125939_56.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/d002acac-1111-4e9a-83fd-70109ad90c7c_isrForCrosstalkSources_1125939_56.stderr parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash c505b7a7fdacb5ab1eb2041a91c28f38 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/31c73926-f8e4-4505-92d3-987c7715594d_isrForCrosstalkSources_1125901_39.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/31c73926-f8e4-4505-92d3-987c7715594d_isrForCrosstalkSources_1125901_39.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 6e653b14f28673e9f39c2d92661173bf parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/0a74a70b-a270-4de1-b515-924cdeeb1277_isrForCrosstalkSources_1125896_19.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/0a74a70b-a270-4de1-b515-924cdeeb1277_isrForCrosstalkSources_1125896_19.stderr parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 1e07de5b1e320785b823fa50427f0329 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/6efdc2d4-d6d5-4712-b45a-3b37b20a68d9_isrForCrosstalkSources_1125914_13.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/6efdc2d4-d6d5-4712-b45a-3b37b20a68d9_isrForCrosstalkSources_1125914_13.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash a68a26a8f4f8908025a09669ba99e26f parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/5325bf4b-4b7b-4056-919b-86decb3f5ec3_isrForCrosstalkSources_1125900_8.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/5325bf4b-4b7b-4056-919b-86decb3f5ec3_isrForCrosstalkSources_1125900_8.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash acef1eb7132e322670788340bc1f0ae3 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/168d01f6-b0c2-443c-b581-33fd5372dfa2_isrForCrosstalkSources_1125936_24.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/168d01f6-b0c2-443c-b581-33fd5372dfa2_isrForCrosstalkSources_1125936_24.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 7ecf36e81f8f914e6af66f8c4b1253c9 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd06fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/8fc8d46f-5578-482d-8ecf-669137467ffe_isrForCrosstalkSources_1125942_48.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/8fc8d46f-5578-482d-8ecf-669137467ffe_isrForCrosstalkSources_1125942_48.stderr parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash e4e064553c08d287b3534d3cd41f4ed0 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/e0026bb8-9de7-4116-a0cd-149ec7d1b8a4_isrForCrosstalkSources_1125915_35.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/e0026bb8-9de7-4116-a0cd-149ec7d1b8a4_isrForCrosstalkSources_1125915_35.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash ac97522edf8f6796e2e8946c96fef476 parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/abf91d4f-d5ab-47c1-907d-6844d3d80361_isrForCrosstalkSources_1125921_54.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/abf91d4f-d5ab-47c1-907d-6844d3d80361_isrForCrosstalkSources_1125921_54.stderr parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 08b21abe007f97ca1550dd30cba98023 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd071a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/607cb5fe-0683-4024-94c6-9ce00bbae060_isrForCrosstalkSources_1125912_5.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/607cb5fe-0683-4024-94c6-9ce00bbae060_isrForCrosstalkSources_1125912_5.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 775f5d50dca95b9a902ba8137e991bcc parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/0812110b-9530-48b5-9ee5-29eecc353bee_isrForCrosstalkSources_1125896_40.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/0812110b-9530-48b5-9ee5-29eecc353bee_isrForCrosstalkSources_1125896_40.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash df1eabf0bb6e6c5cc8b2fe4bf6a4d561 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd072e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/ed81f9e4-61bd-44aa-bc11-889da5883c0d_isrForCrosstalkSources_1125935_39.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/ed81f9e4-61bd-44aa-bc11-889da5883c0d_isrForCrosstalkSources_1125935_39.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash b277d0d3dd966900616dc695fbab8457 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/7c7c2bef-27e7-4ae9-b119-5f98351aaa34_isrForCrosstalkSources_1125943_39.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/7c7c2bef-27e7-4ae9-b119-5f98351aaa34_isrForCrosstalkSources_1125943_39.stderr parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash bd2e80c37e90236e951edf3e88840075 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/6b286417-799c-425e-8dfe-c58530c18e92_isrForCrosstalkSources_1125911_5.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/6b286417-799c-425e-8dfe-c58530c18e92_isrForCrosstalkSources_1125911_5.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash e8bd215ac2d9f5fa8957b9290521a778 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd074c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/80f8f990-da38-49ab-9c27-17871a78f425_isrForCrosstalkSources_1125928_39.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/80f8f990-da38-49ab-9c27-17871a78f425_isrForCrosstalkSources_1125928_39.stderr parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 1b7684d6a6d4c27bbf1c98675cb7b888 parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/2efeb706-cafd-4482-bceb-3e1bb6575d83_isrForCrosstalkSources_1125941_28.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/2efeb706-cafd-4482-bceb-3e1bb6575d83_isrForCrosstalkSources_1125941_28.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 5aca293e182da6011ea46ac261eeb377 parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/73224468-c9a6-47fc-b9e3-dde6fd88e1a4_isrForCrosstalkSources_1125888_55.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/73224468-c9a6-47fc-b9e3-dde6fd88e1a4_isrForCrosstalkSources_1125888_55.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash db2820f239905e7a52bbe76079b7a154 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/4ecee11a-47d1-400d-b98f-a074d0c1c127_isrForCrosstalkSources_1125927_1.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/4ecee11a-47d1-400d-b98f-a074d0c1c127_isrForCrosstalkSources_1125927_1.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 2736feabfdac6ef3c1c19d525c17d8a9 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd076a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/aed14dc1-35e6-4645-afcf-049b7ddfaf54_isrForCrosstalkSources_1125930_61.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/aed14dc1-35e6-4645-afcf-049b7ddfaf54_isrForCrosstalkSources_1125930_61.stderr parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash ff7cc7a007bc159f4b1725f1dec34713 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd077e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/febbaaf8-e91e-4325-bdb4-25d0f96d4846_isrForCrosstalkSources_1125927_45.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/febbaaf8-e91e-4325-bdb4-25d0f96d4846_isrForCrosstalkSources_1125927_45.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash f70d5ab168b600d7c99c99f034b769d6 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/bb9272c3-83fc-4874-8efe-6460b5813de1_isrForCrosstalkSources_1125935_62.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/bb9272c3-83fc-4874-8efe-6460b5813de1_isrForCrosstalkSources_1125935_62.stderr parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 1aab2eef0444ee36425da60fbb243677 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/dc0082f1-178c-442e-8fcb-f5f7f5bec00c_isrForCrosstalkSources_1125911_48.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/dc0082f1-178c-442e-8fcb-f5f7f5bec00c_isrForCrosstalkSources_1125911_48.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 547c8a4216cb93032143747e34edc3a5 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8675e9d7-1dcb-4253-aa22-4f2b944baefa_isrForCrosstalkSources_1125936_38.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8675e9d7-1dcb-4253-aa22-4f2b944baefa_isrForCrosstalkSources_1125936_38.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash d1f008d9f0c9ad8ee377e3ec83e11533 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd079c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/64de8585-672f-4d27-99fb-546a7d737864_isrForCrosstalkSources_1125888_22.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/64de8585-672f-4d27-99fb-546a7d737864_isrForCrosstalkSources_1125888_22.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash c1d19d35d965f6f81246451ceb1d0fba parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b0690202-afa1-4e98-9d00-48f256240cc9_isrForCrosstalkSources_1125934_51.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b0690202-afa1-4e98-9d00-48f256240cc9_isrForCrosstalkSources_1125934_51.stderr parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash fdf9b31bd427ff584c8c60b8eaed683c parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ba0e933c-a86d-4970-8896-9d0cd0b9160b_isrForCrosstalkSources_1125902_55.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ba0e933c-a86d-4970-8896-9d0cd0b9160b_isrForCrosstalkSources_1125902_55.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash e2d53e68e14f8038f39748a966b9b04a parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/28286a84-71c5-482a-a072-4a089290405f_isrForCrosstalkSources_1125911_1.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/28286a84-71c5-482a-a072-4a089290405f_isrForCrosstalkSources_1125911_1.stderr parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash e5b34b71b51202f3fc5f3d9af9437b79 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/9de91466-495e-47fa-ab67-4a9efae2a1c2_isrForCrosstalkSources_1125891_51.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/9de91466-495e-47fa-ab67-4a9efae2a1c2_isrForCrosstalkSources_1125891_51.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash d13f86bd4ef014a7d71b48adca17c269 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/9075aa40-0841-4f95-a580-0816351cc4ed_isrForCrosstalkSources_1125926_20.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/9075aa40-0841-4f95-a580-0816351cc4ed_isrForCrosstalkSources_1125926_20.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash cf313e8c6d4d84a1f72c56b4c3f2fffc parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/8614975c-e20b-4bbc-b5d9-6c2f4e4d63ba_isrForCrosstalkSources_1125920_8.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/8614975c-e20b-4bbc-b5d9-6c2f4e4d63ba_isrForCrosstalkSources_1125920_8.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 041c2686b0a58cbefc8f705ffe02e39b parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/bc8db282-96bf-434c-bc81-75d19b7cc731_isrForCrosstalkSources_1125902_46.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/bc8db282-96bf-434c-bc81-75d19b7cc731_isrForCrosstalkSources_1125902_46.stderr parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 9c4969c8673f4a85a186e080ca3e49b5 parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd07f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/d75e5107-5992-419e-a5c1-8fe48293f49c_isrForCrosstalkSources_1125931_47.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/d75e5107-5992-419e-a5c1-8fe48293f49c_isrForCrosstalkSources_1125931_47.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 2d22a27dd6cb050f632359179dce4af1 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3bd05d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/dae06d16-a9c0-43e3-b974-f7d725ef0275_isrForCrosstalkSources_1125926_15.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/dae06d16-a9c0-43e3-b974-f7d725ef0275_isrForCrosstalkSources_1125926_15.stderr parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash a997fede114fcb666bce3f46f950efe5 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/62c825b9-92d0-4620-8c67-6ad44d3730b3_isrForCrosstalkSources_1125938_11.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/62c825b9-92d0-4620-8c67-6ad44d3730b3_isrForCrosstalkSources_1125938_11.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 970df926b9da5f39f03e9736ae637b4e parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/8592b626-8983-4967-8fe5-453baedb3676_isrForCrosstalkSources_1125917_25.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/8592b626-8983-4967-8fe5-453baedb3676_isrForCrosstalkSources_1125917_25.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash ed9180629fbd6966a4dd5a651c6ae0d1 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0632ac0f-2158-4411-9d7f-133862ff2146_isrForCrosstalkSources_1125904_57.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0632ac0f-2158-4411-9d7f-133862ff2146_isrForCrosstalkSources_1125904_57.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 70bdfe98ea1fc1d2f860a056be74c6fe parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8042c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/54556909-eaab-47c3-a3a2-0ece8cf1e3a3_isrForCrosstalkSources_1125889_19.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/54556909-eaab-47c3-a3a2-0ece8cf1e3a3_isrForCrosstalkSources_1125889_19.stderr parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash d5398a87a31f94780527da17bbc7a392 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/3a1ca9d3-2075-47ba-a1d1-e460b38a03f1_isrForCrosstalkSources_1125917_18.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/3a1ca9d3-2075-47ba-a1d1-e460b38a03f1_isrForCrosstalkSources_1125917_18.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash bd37b23ec9728e9b3ea6f30c51cd2053 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/6795e208-7e09-41b6-acaa-aea6b8a6f407_isrForCrosstalkSources_1125896_38.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/6795e208-7e09-41b6-acaa-aea6b8a6f407_isrForCrosstalkSources_1125896_38.stderr parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash ecb3f97659899f2832d35bba5f612322 parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8044a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/a894a6db-3988-4802-a39a-0938ef572199_isrForCrosstalkSources_1125914_33.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/a894a6db-3988-4802-a39a-0938ef572199_isrForCrosstalkSources_1125914_33.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash e22536481f6abc9c04332fa6e40538e4 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/1d14e3ea-8303-422e-a3a0-e19641cbb05a_isrForCrosstalkSources_1125919_5.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/1d14e3ea-8303-422e-a3a0-e19641cbb05a_isrForCrosstalkSources_1125919_5.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 6b2ccc6d39b4748c8c2b6ec280b86343 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8045e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/289670df-1c85-4ee0-aed2-7d506c292acb_isrForCrosstalkSources_1125937_60.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/289670df-1c85-4ee0-aed2-7d506c292acb_isrForCrosstalkSources_1125937_60.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 069fa67b44531a5c9788a32f018748ba parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/33495f80-52c0-4559-8cb0-d5b44d2d5703_isrForCrosstalkSources_1125922_25.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/33495f80-52c0-4559-8cb0-d5b44d2d5703_isrForCrosstalkSources_1125922_25.stderr parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash b24e9ff304812444a10e558e8893ce4e parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/1beff0ee-ec59-48ce-8ab2-9e25ab3c3b65_isrForCrosstalkSources_1125920_39.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/1beff0ee-ec59-48ce-8ab2-9e25ab3c3b65_isrForCrosstalkSources_1125920_39.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash d87daa789628339f4b9f6575fe5634eb parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8047c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d7033dc8-3829-4179-b9f5-338205e635e0_isrForCrosstalkSources_1125934_5.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d7033dc8-3829-4179-b9f5-338205e635e0_isrForCrosstalkSources_1125934_5.stderr parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 06089d4292f09b18507ed354f513529b parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/714b4f08-0973-4c61-acde-fcb21312debd_isrForCrosstalkSources_1125933_14.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/714b4f08-0973-4c61-acde-fcb21312debd_isrForCrosstalkSources_1125933_14.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash e8b474648b3a0e98ac0427ee36e8eeb5 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/469ffaf9-47c3-48c6-b693-12411826bba0_isrForCrosstalkSources_1125940_56.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/469ffaf9-47c3-48c6-b693-12411826bba0_isrForCrosstalkSources_1125940_56.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 6d66f06f6efb3c310264b82768037d77 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/f76b57a9-7371-4441-b206-4f47022aff9d_isrForCrosstalkSources_1125929_11.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/f76b57a9-7371-4441-b206-4f47022aff9d_isrForCrosstalkSources_1125929_11.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 688672dfa957a896da861e20fd229db4 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8049a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/c2d1416b-c86d-45c0-97e7-c6ac5dd2d3b7_isrForCrosstalkSources_1125925_48.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/c2d1416b-c86d-45c0-97e7-c6ac5dd2d3b7_isrForCrosstalkSources_1125925_48.stderr parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash bd5b61f82652886690c76923c3c9683b parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/a0667ead-71d8-4c0e-b6ae-6470e6a08150_isrForCrosstalkSources_1125947_7.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/a0667ead-71d8-4c0e-b6ae-6470e6a08150_isrForCrosstalkSources_1125947_7.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 9764454b0ce84319645c909833180e18 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/88ba8e88-e291-4a53-be42-546245aeaf2e_isrForCrosstalkSources_1125912_45.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/88ba8e88-e291-4a53-be42-546245aeaf2e_isrForCrosstalkSources_1125912_45.stderr parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash e7615302f3a2279ca832f1a6df4c1898 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/366a27dc-c939-46ed-89f1-3b26405fca74_isrForCrosstalkSources_1125909_46.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/366a27dc-c939-46ed-89f1-3b26405fca74_isrForCrosstalkSources_1125909_46.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash c69d08c5373250c5e183d03e7d11a4b8 parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/9fb8f3f4-6b30-4657-9e9c-ca715369af0c_isrForCrosstalkSources_1125947_18.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/9fb8f3f4-6b30-4657-9e9c-ca715369af0c_isrForCrosstalkSources_1125947_18.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash be4f26bc53b20ec7995e9301c06ad182 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/3705d228-d240-4bf6-982c-fc43ed5aa233_isrForCrosstalkSources_1125927_24.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/3705d228-d240-4bf6-982c-fc43ed5aa233_isrForCrosstalkSources_1125927_24.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash 1ad2feb878c9fa6fae30db02569cfed6 parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/a8a97aa1-f76b-43d6-91d4-83601e208319_isrForCrosstalkSources_1125906_19.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/a8a97aa1-f76b-43d6-91d4-83601e208319_isrForCrosstalkSources_1125906_19.stderr parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash ef857a8b370b645c7db6792abf9cbd15 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/1f635357-0ce4-40f5-996f-a2324e877a50_isrForCrosstalkSources_1125938_40.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/1f635357-0ce4-40f5-996f-a2324e877a50_isrForCrosstalkSources_1125938_40.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 31b5d7602de4ec5827760f9527e51b8c parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e29dc6fa-6f9d-44b4-8cdc-a7c71f3cfc35_isrForCrosstalkSources_1125895_10.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e29dc6fa-6f9d-44b4-8cdc-a7c71f3cfc35_isrForCrosstalkSources_1125895_10.stderr parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 7e631f4d037b07b6c5d3545b601ecd40 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/62ea93e6-9cd2-43e7-aa45-96440f82c2bc_isrForCrosstalkSources_1125890_35.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/62ea93e6-9cd2-43e7-aa45-96440f82c2bc_isrForCrosstalkSources_1125890_35.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash cc7899e542f7034a526b4ecf197acb55 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c804fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/bbb5b755-f86b-4b56-9212-e74c026de731_isrForCrosstalkSources_1125917_56.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/bbb5b755-f86b-4b56-9212-e74c026de731_isrForCrosstalkSources_1125917_56.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash c642d84df6aade5a5c4f8d7cb7683052 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/72ecf0af-23e9-4dfb-8439-ae051d55721e_isrForCrosstalkSources_1125906_33.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/72ecf0af-23e9-4dfb-8439-ae051d55721e_isrForCrosstalkSources_1125906_33.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash b9cf4f2b25d087268290417f19e5db3d parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8051c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/8a4df76e-5fae-496a-adf3-66c29d1f96a1_isrForCrosstalkSources_1125888_6.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/8a4df76e-5fae-496a-adf3-66c29d1f96a1_isrForCrosstalkSources_1125888_6.stderr parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash bdd7a945822642b45e4a12a7f62f0370 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0b06d4dd-bad0-4db8-a0a8-aaef9eb0f455_isrForCrosstalkSources_1125914_36.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0b06d4dd-bad0-4db8-a0a8-aaef9eb0f455_isrForCrosstalkSources_1125914_36.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 5105d77466551aa93122cf4a6e91052d parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8053a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/63305e4a-1bb0-4f25-86ce-7d38ea7d08d8_isrForCrosstalkSources_1125893_33.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/63305e4a-1bb0-4f25-86ce-7d38ea7d08d8_isrForCrosstalkSources_1125893_33.stderr parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash 51956aa3e19726e01986e433e739d1be parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/4d52f08f-8321-42e2-ae82-a3364e154af7_isrForCrosstalkSources_1125910_61.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/4d52f08f-8321-42e2-ae82-a3364e154af7_isrForCrosstalkSources_1125910_61.stderr parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash c0a9350eeaf6ebe2ae44ff91a7847b41 parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/0412de71-72b8-4040-b53b-cd71aa4cc01a_isrForCrosstalkSources_1125889_58.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/0412de71-72b8-4040-b53b-cd71aa4cc01a_isrForCrosstalkSources_1125889_58.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 5d3c50676a9a6bddfdf8ce9936bd3953 parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8054e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/cd7b2219-ea0f-443a-92b3-8d7e186ba01b_isrForCrosstalkSources_1125906_59.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/cd7b2219-ea0f-443a-92b3-8d7e186ba01b_isrForCrosstalkSources_1125906_59.stderr parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 61a1dd722c6c03b855faedfdc5114931 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/2f41f274-a393-47fb-b8e1-54b55698c23b_isrForCrosstalkSources_1125908_62.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/2f41f274-a393-47fb-b8e1-54b55698c23b_isrForCrosstalkSources_1125908_62.stderr parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 3a4eef9970a44d6a98c104722d8e4de0 parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8056c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3b3c7554-bfef-4512-98ea-729c71419982_isrForCrosstalkSources_1125920_10.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3b3c7554-bfef-4512-98ea-729c71419982_isrForCrosstalkSources_1125920_10.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 3959dbdbec1a3aa78df00b2af279f322 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/6f71ab4a-897b-443f-aa15-871a55cba00e_isrForCrosstalkSources_1125907_23.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/6f71ab4a-897b-443f-aa15-871a55cba00e_isrForCrosstalkSources_1125907_23.stderr parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash a825bd268b5e98bcd6860000a0446ea5 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/c3fd174b-ba59-4845-bc3a-a4403d553fca_isrForCrosstalkSources_1125895_59.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/c3fd174b-ba59-4845-bc3a-a4403d553fca_isrForCrosstalkSources_1125895_59.stderr parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash a8a14aded3a0571cb471d65d911a1897 parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/3854e754-046c-4f5f-a16f-165a0b16071e_isrForCrosstalkSources_1125894_62.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/3854e754-046c-4f5f-a16f-165a0b16071e_isrForCrosstalkSources_1125894_62.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash e47a6f140a8c1738246771a635016f9c parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8058a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/29d49340-ac61-480b-848f-76f9cd85e1c4_isrForCrosstalkSources_1125937_21.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/29d49340-ac61-480b-848f-76f9cd85e1c4_isrForCrosstalkSources_1125937_21.stderr parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 8d52d60a34da3d7b3b7391aa98bf139d parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8059e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/25a3a2fe-d705-4bf9-aa68-e861b114a253_isrForCrosstalkSources_1125919_59.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/25a3a2fe-d705-4bf9-aa68-e861b114a253_isrForCrosstalkSources_1125919_59.stderr parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash b06d67dc72de3da77765550e4b1a4a1c parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/0e64442e-5319-4a1d-8e2d-a9ab2971b13a_isrForCrosstalkSources_1125917_60.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/0e64442e-5319-4a1d-8e2d-a9ab2971b13a_isrForCrosstalkSources_1125917_60.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 3b9b73a21a7c5e4d506553671469a4cd parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ff308b62-8263-42a3-9c17-eaf1680f4cd1_isrForCrosstalkSources_1125902_16.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ff308b62-8263-42a3-9c17-eaf1680f4cd1_isrForCrosstalkSources_1125902_16.stderr parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 8df181076b328706ae3ac181baad6b2c parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/ade256fd-ee0f-4379-9f84-59ee4702c62f_isrForCrosstalkSources_1125937_56.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/ade256fd-ee0f-4379-9f84-59ee4702c62f_isrForCrosstalkSources_1125937_56.stderr parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 80bbfa09475caa4ca80bbe35f0c1c042 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/a0cbc1d6-3f9d-456b-a0e9-9040738f36ab_isrForCrosstalkSources_1125890_37.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/a0cbc1d6-3f9d-456b-a0e9-9040738f36ab_isrForCrosstalkSources_1125890_37.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash c8a0e8b0cf3d142c4146c16337940b2c parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/c6f1291c-d0a4-4364-8615-39091836c6dd_isrForCrosstalkSources_1125905_43.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/c6f1291c-d0a4-4364-8615-39091836c6dd_isrForCrosstalkSources_1125905_43.stderr parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 2a67f76efce23700a00b1d2cd98382fe parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/f60ff89d-1205-4d89-9cae-8966bee255a8_isrForCrosstalkSources_1125931_43.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/f60ff89d-1205-4d89-9cae-8966bee255a8_isrForCrosstalkSources_1125931_43.stderr parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 9b5ccb290f903f0d27c4ca8acf356abb parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/15abafaf-ecd5-422c-9659-74a3584410c8_isrForCrosstalkSources_1125893_29.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/15abafaf-ecd5-422c-9659-74a3584410c8_isrForCrosstalkSources_1125893_29.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 8fc055de21dc1ca5dde54c237f560d0a parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f138958b-0e82-4564-bcdd-5715508356d2_isrForCrosstalkSources_1125939_44.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f138958b-0e82-4564-bcdd-5715508356d2_isrForCrosstalkSources_1125939_44.stderr parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash b59e724cfdfac60dcbc75e1feabfb081 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/666adf21-561a-40ca-ad45-cb5f77dc3bcb_isrForCrosstalkSources_1125930_29.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/666adf21-561a-40ca-ad45-cb5f77dc3bcb_isrForCrosstalkSources_1125930_29.stderr parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 92b1bd1b37954fbe29ef82af12c9dc88 parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/81f53f79-b7e0-44cf-8b98-659fc2485b9e_isrForCrosstalkSources_1125903_5.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/81f53f79-b7e0-44cf-8b98-659fc2485b9e_isrForCrosstalkSources_1125903_5.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 58b36801d08328c4f7b5bc3f6552db8a parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c805f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/22c2dda9-1ebd-41aa-a3cd-21715b9134df_isrForCrosstalkSources_1125915_25.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/22c2dda9-1ebd-41aa-a3cd-21715b9134df_isrForCrosstalkSources_1125915_25.stderr parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 2ecca634d6a66165f76b132a2122b39c parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8060c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/bb84629b-f8dd-4b94-b4c9-d3bfb385656a_isrForCrosstalkSources_1125913_42.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/bb84629b-f8dd-4b94-b4c9-d3bfb385656a_isrForCrosstalkSources_1125913_42.stderr parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 3f0689e558cb0d345fece4fe6dc1b5ce parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/9ebdaf25-5e4f-4765-bdfd-c95ac7a506ea_isrForCrosstalkSources_1125923_12.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/9ebdaf25-5e4f-4765-bdfd-c95ac7a506ea_isrForCrosstalkSources_1125923_12.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 54a2a222a1f0f70930ae789ef4d75741 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/cf31fd17-9857-49e7-98be-022e5af5d2ce_isrForCrosstalkSources_1125918_14.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/cf31fd17-9857-49e7-98be-022e5af5d2ce_isrForCrosstalkSources_1125918_14.stderr parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash cf22df2e3530bdfa02ee4aa304e400a8 parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/af1088cf-c36d-4e4a-881b-e769f12c2fc3_isrForCrosstalkSources_1125904_15.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/af1088cf-c36d-4e4a-881b-e769f12c2fc3_isrForCrosstalkSources_1125904_15.stderr parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 44fb8eda153140ea3b80e369e82802e7 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8062a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/0abb1be5-113a-418c-84c2-48efd05115df_isrForCrosstalkSources_1125918_37.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/0abb1be5-113a-418c-84c2-48efd05115df_isrForCrosstalkSources_1125918_37.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 37587f86a922f3831620c9b633ecfb4e parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8063e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/3c911783-249c-4786-9d68-6b676ab56601_isrForCrosstalkSources_1125888_7.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/3c911783-249c-4786-9d68-6b676ab56601_isrForCrosstalkSources_1125888_7.stderr parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 8b158bd8b9d2026fbf02c5c128d31957 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/2a026a11-9fa9-4b31-a685-3f3551b4a8b0_isrForCrosstalkSources_1125892_43.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/2a026a11-9fa9-4b31-a685-3f3551b4a8b0_isrForCrosstalkSources_1125892_43.stderr parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash e0fb6a4c8677e767beaba6121c82770f parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/a8cb95fa-4925-4251-8878-7c24255f4f8f_isrForCrosstalkSources_1125920_28.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/a8cb95fa-4925-4251-8878-7c24255f4f8f_isrForCrosstalkSources_1125920_28.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 53db8c39a0f4bab4fe1884c74a549830 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/1c503253-69ad-4790-8825-05e5324ec5c1_isrForCrosstalkSources_1125926_23.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/1c503253-69ad-4790-8825-05e5324ec5c1_isrForCrosstalkSources_1125926_23.stderr parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 9d0e0c432a5ab730a630c1bb921b9c3a parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8065c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/8d7c1e32-928a-4d4f-b4c4-fca2b62c17ad_isrForCrosstalkSources_1125930_32.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/8d7c1e32-928a-4d4f-b4c4-fca2b62c17ad_isrForCrosstalkSources_1125930_32.stderr parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 1c5f2ef086a4902a88c37fbcae6f604b parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/6f0d18e2-2020-48ae-aa65-47022310fcc0_isrForCrosstalkSources_1125924_57.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/6f0d18e2-2020-48ae-aa65-47022310fcc0_isrForCrosstalkSources_1125924_57.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 7a7101c8d207024eed8dc9a51684895c parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8067a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/ccb1a9da-6276-4262-8275-459c3c0a4f10_isrForCrosstalkSources_1125920_40.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/ccb1a9da-6276-4262-8275-459c3c0a4f10_isrForCrosstalkSources_1125920_40.stderr parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash fb54157114dc9ee892bec400564a09a9 parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/15fb4bca-209f-47e6-b9e0-45ca3de7831e_isrForCrosstalkSources_1125920_3.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/15fb4bca-209f-47e6-b9e0-45ca3de7831e_isrForCrosstalkSources_1125920_3.stderr parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 5080d887dd0d7289556aa630db6073d1 parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/826ac9c7-e63c-47c0-837a-7f3ce6078437_isrForCrosstalkSources_1125945_15.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/826ac9c7-e63c-47c0-837a-7f3ce6078437_isrForCrosstalkSources_1125945_15.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash 68e5a7131402e0358c3eff29ae12a397 parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8068e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/01552f8b-a2d3-48aa-9964-c394c983bd06_isrForCrosstalkSources_1125907_51.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/01552f8b-a2d3-48aa-9964-c394c983bd06_isrForCrosstalkSources_1125907_51.stderr parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 00c735df0e568245884900373ead3681 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/c75be3f1-6c97-46a9-bfdf-016b59b88cdd_isrForCrosstalkSources_1125911_55.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/c75be3f1-6c97-46a9-bfdf-016b59b88cdd_isrForCrosstalkSources_1125911_55.stderr parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 167aa4793691c1e82834ece064d31589 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/e3bb5c17-3402-481c-a0c0-217ca7a08141_isrForCrosstalkSources_1125902_29.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/e3bb5c17-3402-481c-a0c0-217ca7a08141_isrForCrosstalkSources_1125902_29.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash acd992e7e6e06d3e1e35c84fc1d06f4b parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/d2bb05e0-0711-43a1-85c4-93955c2b051b_isrForCrosstalkSources_1125916_52.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/d2bb05e0-0711-43a1-85c4-93955c2b051b_isrForCrosstalkSources_1125916_52.stderr parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash c9c0974f704f37bf3938f1503a9f3f6d parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/698263a9-86a4-4152-a379-69941e3197b1_isrForCrosstalkSources_1125909_26.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/698263a9-86a4-4152-a379-69941e3197b1_isrForCrosstalkSources_1125909_26.stderr parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 84e41e318c73339a4185932ec762631e parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/beed4d6a-efbc-4879-b909-9b5e676abe60_isrForCrosstalkSources_1125937_54.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/beed4d6a-efbc-4879-b909-9b5e676abe60_isrForCrosstalkSources_1125937_54.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 27b8b70c1e9451ac934f65b18053cded parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/e7d37021-b7b0-46c6-b8ce-93b5fad38bf5_isrForCrosstalkSources_1125910_10.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/e7d37021-b7b0-46c6-b8ce-93b5fad38bf5_isrForCrosstalkSources_1125910_10.stderr parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 9a0f64e38cb206383aa22e33d2f315ad parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/69c42ab2-fa69-4fda-b4ec-874d1caf6376_isrForCrosstalkSources_1125912_14.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/69c42ab2-fa69-4fda-b4ec-874d1caf6376_isrForCrosstalkSources_1125912_14.stderr parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 7533e987c4cc7fd39a027a97641af906 parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/9a5a5a71-3420-47d5-8984-012cea19a398_isrForCrosstalkSources_1125889_51.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/9a5a5a71-3420-47d5-8984-012cea19a398_isrForCrosstalkSources_1125889_51.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash fac423ff93fc08e3c59973c0d4d89beb parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/ba06c33e-6b8c-4fc9-982f-7b4f6491c3f0_isrForCrosstalkSources_1125915_47.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/ba06c33e-6b8c-4fc9-982f-7b4f6491c3f0_isrForCrosstalkSources_1125915_47.stderr parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 97ee712b347c12d95e6684596d0b1032 parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c806f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/2b4917aa-26d9-4ae0-a8fa-0b57732e0b5d_isrForCrosstalkSources_1125932_1.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/2b4917aa-26d9-4ae0-a8fa-0b57732e0b5d_isrForCrosstalkSources_1125932_1.stderr parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 0f30ae3569fabc2f18da065a1af33f97 parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/b9ded6a8-c3a0-47aa-b4de-56edf6e784ed_isrForCrosstalkSources_1125899_61.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/b9ded6a8-c3a0-47aa-b4de-56edf6e784ed_isrForCrosstalkSources_1125899_61.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 2c1e5dda81314afa05e88b29002b2f0a parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/ea3ee25b-2ef5-4cc0-aab8-f91800da7c34_isrForCrosstalkSources_1125904_10.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/ea3ee25b-2ef5-4cc0-aab8-f91800da7c34_isrForCrosstalkSources_1125904_10.stderr parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 14d410712a6a2cb0e1d67e1ef8c93535 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8071a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/7986f352-8146-456f-9fc8-61581f1516dd_isrForCrosstalkSources_1125925_55.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/7986f352-8146-456f-9fc8-61581f1516dd_isrForCrosstalkSources_1125925_55.stderr parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 4391ab60c41db4091e73e0127b435cf3 parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8072e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/85f506b7-e359-4613-aac0-cfcf856bb1ba_isrForCrosstalkSources_1125938_22.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/85f506b7-e359-4613-aac0-cfcf856bb1ba_isrForCrosstalkSources_1125938_22.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 6180ad2ba0ea43967278512a91ac4822 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/61f6ae8a-c2f1-4cb8-8ea6-21d7f1d7dc9f_isrForCrosstalkSources_1125944_34.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/61f6ae8a-c2f1-4cb8-8ea6-21d7f1d7dc9f_isrForCrosstalkSources_1125944_34.stderr parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 9e0b5b25e353817c59deb04b925cdc72 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/4fc298f7-8e75-4681-8211-b1a75387ae5c_isrForCrosstalkSources_1125894_50.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/4fc298f7-8e75-4681-8211-b1a75387ae5c_isrForCrosstalkSources_1125894_50.stderr parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash cffbc6cd89ef776bd6fcdd2fe393a625 parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/56fc3d9c-ac57-45ec-ae7c-db6fc9271302_isrForCrosstalkSources_1125896_18.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/56fc3d9c-ac57-45ec-ae7c-db6fc9271302_isrForCrosstalkSources_1125896_18.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 32c5e4bf3210dd3aa47ae397295abf57 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8074c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/cd410c97-5f89-45de-a111-7858bd4c1357_isrForCrosstalkSources_1125931_23.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/cd410c97-5f89-45de-a111-7858bd4c1357_isrForCrosstalkSources_1125931_23.stderr parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash ab2a219e04112f753fa2b377160605f8 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/355c6df0-0e71-4c70-a37a-71888d0fdeed_isrForCrosstalkSources_1125916_40.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/355c6df0-0e71-4c70-a37a-71888d0fdeed_isrForCrosstalkSources_1125916_40.stderr parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 21d93193ab4a36ded19ab6aee1ba645c parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/ca30b558-0a71-41d9-b3ef-dbe1ba303381_isrForCrosstalkSources_1125946_55.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/ca30b558-0a71-41d9-b3ef-dbe1ba303381_isrForCrosstalkSources_1125946_55.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 37e2a1d64b2d7ef000285c33fd81f09d parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8076a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/d730323a-6051-4b50-b394-36340b9a47c8_isrForCrosstalkSources_1125944_12.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/d730323a-6051-4b50-b394-36340b9a47c8_isrForCrosstalkSources_1125944_12.stderr parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 2806cd6a7ca63927a2f1645eb2788c67 parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/14a91392-0e3b-4440-97ac-44c10c1b8083_isrForCrosstalkSources_1125898_59.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/14a91392-0e3b-4440-97ac-44c10c1b8083_isrForCrosstalkSources_1125898_59.stderr parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 3bcb4c5fee105dd65a86a43a50c3c1fc parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8077e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/16db3937-bede-412f-8ded-ac443020390a_isrForCrosstalkSources_1125909_35.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/16db3937-bede-412f-8ded-ac443020390a_isrForCrosstalkSources_1125909_35.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash a4b25122cccb34e0a8b514adff29de53 parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/1f18e146-c162-4bfd-accf-8c6c6d748298_isrForCrosstalkSources_1125947_33.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/1f18e146-c162-4bfd-accf-8c6c6d748298_isrForCrosstalkSources_1125947_33.stderr parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 0704dcaca649aabcfede1c35425d5ece parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/6d216ce8-7ff8-4e92-aac5-1f7f62dacdde_isrForCrosstalkSources_1125930_30.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/6d216ce8-7ff8-4e92-aac5-1f7f62dacdde_isrForCrosstalkSources_1125930_30.stderr parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 8c4ff199841474961eb6de88783d5adb parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8079c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/c1696cfb-89ed-4972-b6f3-c5b0bb60d6ca_isrForCrosstalkSources_1125890_3.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/c1696cfb-89ed-4972-b6f3-c5b0bb60d6ca_isrForCrosstalkSources_1125890_3.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 2eac40d4719c947cd7786a478e121f92 parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/a554d517-599c-4c1c-86f7-32ea9bce6185_isrForCrosstalkSources_1125903_27.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/a554d517-599c-4c1c-86f7-32ea9bce6185_isrForCrosstalkSources_1125903_27.stderr parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 99b13eca51dc371a8f8a15adfc6b4bf5 parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/3458dc8d-6502-4373-92b0-f8a219be66ea_isrForCrosstalkSources_1125888_62.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/3458dc8d-6502-4373-92b0-f8a219be66ea_isrForCrosstalkSources_1125888_62.stderr parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 9810fed7d564e7669563f4816806ed26 parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/2e9837d2-f951-4e56-afa9-9557614c8fd5_isrForCrosstalkSources_1125891_26.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/2e9837d2-f951-4e56-afa9-9557614c8fd5_isrForCrosstalkSources_1125891_26.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 7602c253cf9ad4431a642431f9e1b486 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7c56235d-f73a-4804-ae72-60d791ed8253_isrForCrosstalkSources_1125929_51.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7c56235d-f73a-4804-ae72-60d791ed8253_isrForCrosstalkSources_1125929_51.stderr parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash edc3a1fa6994b6711fa584b6fe2a8ac9 parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/68579b87-737d-4375-b0da-461edbb846d4_isrForCrosstalkSources_1125912_33.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/68579b87-737d-4375-b0da-461edbb846d4_isrForCrosstalkSources_1125912_33.stderr parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 5a89568dd4b9e8e5f1fbfa678242301a parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/7025fb9d-e392-4ee4-93a9-968dbc1a2b5c_isrForCrosstalkSources_1125936_21.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/7025fb9d-e392-4ee4-93a9-968dbc1a2b5c_isrForCrosstalkSources_1125936_21.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 6d2468c54cbbc122434fbffd7869d6dc parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/3bbe0d0a-5266-455a-ad16-31e7ac2af7ba_isrForCrosstalkSources_1125899_13.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/3bbe0d0a-5266-455a-ad16-31e7ac2af7ba_isrForCrosstalkSources_1125899_13.stderr parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 4d4af41e5ea5c8f825f347604a01efd3 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/d5bd26bd-e3c8-498e-b4e7-1f56d0aff5f4_isrForCrosstalkSources_1125931_25.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/d5bd26bd-e3c8-498e-b4e7-1f56d0aff5f4_isrForCrosstalkSources_1125931_25.stderr parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash be3a8fa0b6dfa1485c005042dc497891 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c807ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/f67ddc4b-c7cc-4407-a359-9284b3a92228_isrForCrosstalkSources_1125926_39.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/f67ddc4b-c7cc-4407-a359-9284b3a92228_isrForCrosstalkSources_1125926_39.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 2b581f06fbbfffc716765b9c9aed9106 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c8040e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/ce27b0a0-5131-4300-ad7d-8134bfdcaa6c_isrForCrosstalkSources_1125927_5.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/ce27b0a0-5131-4300-ad7d-8134bfdcaa6c_isrForCrosstalkSources_1125927_5.stderr parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash d7b5c83e09b3cf4b29d952b2ef24a828 parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9940e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/8644772b-8920-4091-a02a-cdb5e0e1ddf7_isrForCrosstalkSources_1125922_28.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/8644772b-8920-4091-a02a-cdb5e0e1ddf7_isrForCrosstalkSources_1125922_28.stderr parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 8e34f8e336b6edf5eb6c719fbde96537 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/d0bff332-717c-487d-826b-e8c9a6acfe87_isrForCrosstalkSources_1125909_59.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/d0bff332-717c-487d-826b-e8c9a6acfe87_isrForCrosstalkSources_1125909_59.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash d33afa27997fa87e206eaaa6b4db54b7 parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9942c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/dfddd713-fc13-4a29-aa9e-0e4223077122_isrForCrosstalkSources_1125943_34.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/dfddd713-fc13-4a29-aa9e-0e4223077122_isrForCrosstalkSources_1125943_34.stderr parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 2fd839d015a6176ef9c136ac65909719 parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/2bc976ad-c093-4887-9ba2-ab0ef6a45e67_isrForCrosstalkSources_1125927_2.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/2bc976ad-c093-4887-9ba2-ab0ef6a45e67_isrForCrosstalkSources_1125927_2.stderr parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 9c922feb61b5918b6ba781352cbd2d00 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/654405c9-4952-4c6b-8c11-5f2771cbff3b_isrForCrosstalkSources_1125892_27.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/654405c9-4952-4c6b-8c11-5f2771cbff3b_isrForCrosstalkSources_1125892_27.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 920491c9335ce455c085ecc87f696d0d parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/6d598cfa-7e7a-4c34-9cbc-5bfa99b600f1_isrForCrosstalkSources_1125920_13.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/6d598cfa-7e7a-4c34-9cbc-5bfa99b600f1_isrForCrosstalkSources_1125920_13.stderr parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash cbe4b4b6c6ec79dae32b9ffcbd44f408 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9944a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/523fd886-4c55-4160-b767-10f516889799_isrForCrosstalkSources_1125923_26.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/523fd886-4c55-4160-b767-10f516889799_isrForCrosstalkSources_1125923_26.stderr parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 431fa432747336cc32a6a51485b62b1b parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9945e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/e2fd778e-e8b8-47a5-b705-1c225f76054d_isrForCrosstalkSources_1125910_31.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/e2fd778e-e8b8-47a5-b705-1c225f76054d_isrForCrosstalkSources_1125910_31.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash ce4c68344ff36307be5f165b9443e73d parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/dd713626-1130-406e-b555-a62ff24aff09_isrForCrosstalkSources_1125944_15.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/dd713626-1130-406e-b555-a62ff24aff09_isrForCrosstalkSources_1125944_15.stderr parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash ac2dc039b87b788396afd03406a5c7bd parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/3a324b9b-7747-4fee-9c36-1e58b2a98338_isrForCrosstalkSources_1125930_53.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/3a324b9b-7747-4fee-9c36-1e58b2a98338_isrForCrosstalkSources_1125930_53.stderr parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 6a9b44561214c0753b358199e8a03403 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/c8e86578-fd0f-4001-91eb-7c05e490e280_isrForCrosstalkSources_1125895_3.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/c8e86578-fd0f-4001-91eb-7c05e490e280_isrForCrosstalkSources_1125895_3.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash e4da48c37389cafc062899edfd4acd8c parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9947c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/23308920-2a8f-450b-a24e-62a20a0eda5a_isrForCrosstalkSources_1125889_41.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/23308920-2a8f-450b-a24e-62a20a0eda5a_isrForCrosstalkSources_1125889_41.stderr parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash 66d1ab568232917b5b9c25ab5e21dd83 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/eed76df3-448f-429d-a318-e51840514d9b_isrForCrosstalkSources_1125937_59.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/eed76df3-448f-429d-a318-e51840514d9b_isrForCrosstalkSources_1125937_59.stderr parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash ff1e281e83f608780dbe82082ea5213a parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d9687063-7999-47a6-b7d3-b1e35c182557_isrForCrosstalkSources_1125934_4.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d9687063-7999-47a6-b7d3-b1e35c182557_isrForCrosstalkSources_1125934_4.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 30df9059112dd4c0fa8b9c0b1a1e8a62 parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9949a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/79638e92-6ef0-4c2e-92db-c02312d9c133_isrForCrosstalkSources_1125927_20.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/79638e92-6ef0-4c2e-92db-c02312d9c133_isrForCrosstalkSources_1125927_20.stderr parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash a6b4e66b15b5c4763d8ac29a3c5d6dfc parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/5440665e-1b8c-44f9-8313-c2b34e619ae8_isrForCrosstalkSources_1125902_45.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/5440665e-1b8c-44f9-8313-c2b34e619ae8_isrForCrosstalkSources_1125902_45.stderr parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash c4ceb931cb4c7b75c1370917096fe167 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/047eea61-46de-474a-9a66-c85daeb76876_isrForCrosstalkSources_1125896_30.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/047eea61-46de-474a-9a66-c85daeb76876_isrForCrosstalkSources_1125896_30.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 2c1aa7ad8c14b3cfa85691414110e11c parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/ff35a4c2-b774-41ce-ac22-760eb239b416_isrForCrosstalkSources_1125890_33.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/ff35a4c2-b774-41ce-ac22-760eb239b416_isrForCrosstalkSources_1125890_33.stderr parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 8bcaeacb9018ed8536996f781fa9e2cb parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/edf50b99-c192-469a-a11c-6ff783efadbc_isrForCrosstalkSources_1125946_7.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/edf50b99-c192-469a-a11c-6ff783efadbc_isrForCrosstalkSources_1125946_7.stderr parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 8678a4be08ae04bd9c632d873c001652 parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/a017c7b3-50a2-4af4-9984-cb885403ab0b_isrForCrosstalkSources_1125922_10.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/a017c7b3-50a2-4af4-9984-cb885403ab0b_isrForCrosstalkSources_1125922_10.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 62761bb726a8e316af4a229a42140388 parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/90ab43ac-bd1b-4e39-a509-b5ecc93550e7_isrForCrosstalkSources_1125921_28.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/90ab43ac-bd1b-4e39-a509-b5ecc93550e7_isrForCrosstalkSources_1125921_28.stderr parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash fc9a05a1add02a112822aaf282e8ded1 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/7119a57d-406b-4557-9639-0bd94f679800_isrForCrosstalkSources_1125919_6.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/7119a57d-406b-4557-9639-0bd94f679800_isrForCrosstalkSources_1125919_6.stderr parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash b570b1bb88ffffb3703d1860b5500d4e parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/8c9441c1-0326-4ab8-aea0-ebe88cd60cf8_isrForCrosstalkSources_1125922_42.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/8c9441c1-0326-4ab8-aea0-ebe88cd60cf8_isrForCrosstalkSources_1125922_42.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 715e668e35ff137b46078985661f131a parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/40a730c3-6c3a-4bed-90ec-03bb5d082366_isrForCrosstalkSources_1125942_15.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/40a730c3-6c3a-4bed-90ec-03bb5d082366_isrForCrosstalkSources_1125942_15.stderr parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 68b6261d061508caeccde77cd1ee6890 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/cbe82e7c-9899-4d37-9411-12d124ddf8f4_isrForCrosstalkSources_1125913_46.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/cbe82e7c-9899-4d37-9411-12d124ddf8f4_isrForCrosstalkSources_1125913_46.stderr parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash ffea652a04efb51ee62ec10bfaa3f733 parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/8d5bbef2-d998-49fa-8544-d1a3d18a72ed_isrForCrosstalkSources_1125925_45.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/8d5bbef2-d998-49fa-8544-d1a3d18a72ed_isrForCrosstalkSources_1125925_45.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 9274a7191e5b5d4c394426c2c7bdbee3 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/6b6a8fd5-a4f7-496c-8401-9c4d85cd3767_isrForCrosstalkSources_1125898_17.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/6b6a8fd5-a4f7-496c-8401-9c4d85cd3767_isrForCrosstalkSources_1125898_17.stderr parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash becaf2a93f0d4998e086a485c53d3e29 parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/b5d69f2b-bc2b-40b2-8524-c313df353d24_isrForCrosstalkSources_1125891_53.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/b5d69f2b-bc2b-40b2-8524-c313df353d24_isrForCrosstalkSources_1125891_53.stderr parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 28e4dde88d68dbd50c0dcee2da89e603 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9951c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/e65e7077-8aa7-4cfa-8e65-5958e30a7adb_isrForCrosstalkSources_1125931_50.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/e65e7077-8aa7-4cfa-8e65-5958e30a7adb_isrForCrosstalkSources_1125931_50.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 7e9a0ef1096b0f6b4159d58eae199189 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/8a55f53a-257c-489d-9c4a-664d0e8e1dde_isrForCrosstalkSources_1125928_59.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/8a55f53a-257c-489d-9c4a-664d0e8e1dde_isrForCrosstalkSources_1125928_59.stderr parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash fc645bfe32cbe0e523f1719c63a973e4 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9953a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/b8d5e294-091f-42b9-b9ba-25b194e72cf3_isrForCrosstalkSources_1125933_22.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/b8d5e294-091f-42b9-b9ba-25b194e72cf3_isrForCrosstalkSources_1125933_22.stderr parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 02ac7c31d89d3126855746f24f750342 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/e7b1cd6f-bad3-48d0-86fc-7609f097bc32_isrForCrosstalkSources_1125911_20.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/e7b1cd6f-bad3-48d0-86fc-7609f097bc32_isrForCrosstalkSources_1125911_20.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash f4e27623bd7e8697aafa83ebde682052 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/6d22c9bf-a9b1-4920-ad28-8223236799cf_isrForCrosstalkSources_1125932_26.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/6d22c9bf-a9b1-4920-ad28-8223236799cf_isrForCrosstalkSources_1125932_26.stderr parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash 973c02c1cda3a2ad4e11709363be72a6 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9954e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/95af21a9-fe6a-49b7-8529-81b07649d511_isrForCrosstalkSources_1125897_13.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/95af21a9-fe6a-49b7-8529-81b07649d511_isrForCrosstalkSources_1125897_13.stderr parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 92486060b0fda475635008989c7c86ae parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/7e95f3c6-2d94-4a34-92eb-b8742fd5712f_isrForCrosstalkSources_1125895_48.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/7e95f3c6-2d94-4a34-92eb-b8742fd5712f_isrForCrosstalkSources_1125895_48.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash eaae787feeb1d4f348f9dff498952fc3 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9956c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/01392e8e-be9d-47ca-98f9-30e4536ba49d_isrForCrosstalkSources_1125918_41.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/01392e8e-be9d-47ca-98f9-30e4536ba49d_isrForCrosstalkSources_1125918_41.stderr parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 64f61bed8f95bc90c53568997179943f parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/05d17200-45ea-4160-9a11-2d17c1ec57f1_isrForCrosstalkSources_1125947_14.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/05d17200-45ea-4160-9a11-2d17c1ec57f1_isrForCrosstalkSources_1125947_14.stderr parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash db66f363bf0c81d8bead17ed9efbc8bf parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9958a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/902e0aad-01ac-45f0-ac9c-607f847b7350_isrForCrosstalkSources_1125911_19.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/902e0aad-01ac-45f0-ac9c-607f847b7350_isrForCrosstalkSources_1125911_19.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash a661083174518e5d8d1e1c63aa352f49 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/72ba23f7-bdfa-46f2-b610-1997f3ebab4a_isrForCrosstalkSources_1125894_18.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/72ba23f7-bdfa-46f2-b610-1997f3ebab4a_isrForCrosstalkSources_1125894_18.stderr parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 755b42349877ca9edd141b1e7e018505 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/3b9cda7d-c3bd-459d-90ff-05f86a18f604_isrForCrosstalkSources_1125946_61.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/3b9cda7d-c3bd-459d-90ff-05f86a18f604_isrForCrosstalkSources_1125946_61.stderr parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 6636c435bc06e5aa041c318bb16fafbc parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9959e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/98f8abed-977b-4e5f-9449-8731fd6e4fdf_isrForCrosstalkSources_1125907_20.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/98f8abed-977b-4e5f-9449-8731fd6e4fdf_isrForCrosstalkSources_1125907_20.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash e5539969519b362a1976812aca4b88f1 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/85bc03a7-cd57-4800-8353-3a25926042de_isrForCrosstalkSources_1125942_58.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/85bc03a7-cd57-4800-8353-3a25926042de_isrForCrosstalkSources_1125942_58.stderr parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 12f9b3acc442f41740cd0eb6f4a163fc parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/bf6525f9-50d7-4bca-85e8-d6306ad16484_isrForCrosstalkSources_1125925_46.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/bf6525f9-50d7-4bca-85e8-d6306ad16484_isrForCrosstalkSources_1125925_46.stderr parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 950d1eadb08e088c95a648a3e03f050a parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/781d3871-abdd-485e-8f9a-b8f35cdadff7_isrForCrosstalkSources_1125926_41.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/781d3871-abdd-485e-8f9a-b8f35cdadff7_isrForCrosstalkSources_1125926_41.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash f9d5168a18897c9a66a251caa984ddea parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/156e3a55-729d-4646-9275-1bec2e9030d0_isrForCrosstalkSources_1125927_40.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/156e3a55-729d-4646-9275-1bec2e9030d0_isrForCrosstalkSources_1125927_40.stderr parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash a93b412efbf9a55dfa85841453610d3c parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/88a4d9da-ee8c-4564-9aa9-9ef48fe617c2_isrForCrosstalkSources_1125892_14.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/88a4d9da-ee8c-4564-9aa9-9ef48fe617c2_isrForCrosstalkSources_1125892_14.stderr parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 57157336fd3a1caa7a1efd9bf38db020 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/f624402e-4183-44e7-b067-a7ddd3a69b91_isrForCrosstalkSources_1125942_19.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/f624402e-4183-44e7-b067-a7ddd3a69b91_isrForCrosstalkSources_1125942_19.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash b47f382d4abad596eb7de4992bca157a parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/e6e72c2d-0b93-476f-93e7-f379e9ce275a_isrForCrosstalkSources_1125892_49.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/e6e72c2d-0b93-476f-93e7-f379e9ce275a_isrForCrosstalkSources_1125892_49.stderr parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash aea759b4fe97a193848b224805bca95d parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/c63c267b-999c-47e2-8161-877383e1c3d4_isrForCrosstalkSources_1125939_28.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/c63c267b-999c-47e2-8161-877383e1c3d4_isrForCrosstalkSources_1125939_28.stderr parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash d8950f5623cc762bb0f667f09f208ef4 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c995f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/43bf59b2-a168-4053-ad49-77084061a06c_isrForCrosstalkSources_1125936_44.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/43bf59b2-a168-4053-ad49-77084061a06c_isrForCrosstalkSources_1125936_44.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash b31d06e3d5fc6eac878fbac3010516ce parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/606a685e-83cf-4c6d-98ab-577e6f3a3f96_isrForCrosstalkSources_1125918_28.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/606a685e-83cf-4c6d-98ab-577e6f3a3f96_isrForCrosstalkSources_1125918_28.stderr parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 28d80df5131a19f0d7dea8392116cc50 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9960c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/67a6e94c-69e0-418c-878c-c6ac63bc369e_isrForCrosstalkSources_1125906_57.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/67a6e94c-69e0-418c-878c-c6ac63bc369e_isrForCrosstalkSources_1125906_57.stderr parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash 26839d5870f029f24c05914d3e744030 parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/50294325-8a91-4d1b-a685-14cfa41e5a30_isrForCrosstalkSources_1125891_55.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/50294325-8a91-4d1b-a685-14cfa41e5a30_isrForCrosstalkSources_1125891_55.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash 4a2e840247f42c0b27b69b887a75e1d6 parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b9746f55-47da-4e0f-b247-f4692a8d5277_isrForCrosstalkSources_1125938_6.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b9746f55-47da-4e0f-b247-f4692a8d5277_isrForCrosstalkSources_1125938_6.stderr parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash d39b9b2c2384fad26848c1482af7e3e7 parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9962a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/7b675d7d-f29e-479a-8305-60a62dada638_isrForCrosstalkSources_1125902_36.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/7b675d7d-f29e-479a-8305-60a62dada638_isrForCrosstalkSources_1125902_36.stderr parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash 45d0766ed1815e9c20a7effd5f54e445 parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/3565cdbe-9a7e-491f-83be-f5ce3b561a22_isrForCrosstalkSources_1125901_60.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/3565cdbe-9a7e-491f-83be-f5ce3b561a22_isrForCrosstalkSources_1125901_60.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash 3b140e698f34e69cdd2300039a5e79df parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9963e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/8a9c0b13-b389-4e98-be5c-a2242e842e48_isrForCrosstalkSources_1125905_45.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/8a9c0b13-b389-4e98-be5c-a2242e842e48_isrForCrosstalkSources_1125905_45.stderr parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash 3a51a972b32f3f8daf459bab2bf5fcc6 parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/51db8048-b99d-4e4b-a2c2-647ebd3df0c5_isrForCrosstalkSources_1125924_22.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/51db8048-b99d-4e4b-a2c2-647ebd3df0c5_isrForCrosstalkSources_1125924_22.stderr parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash c6f90843cef5e76ef4a888ddcfae4db1 parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/daed6dbe-4ea4-4542-a5c2-02b9720498ee_isrForCrosstalkSources_1125910_1.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/daed6dbe-4ea4-4542-a5c2-02b9720498ee_isrForCrosstalkSources_1125910_1.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 53958e529bc190f16a5c83d6030d34f7 parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9965c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/493d18ec-f983-4063-a972-3ebcf3ac800b_isrForCrosstalkSources_1125898_23.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/493d18ec-f983-4063-a972-3ebcf3ac800b_isrForCrosstalkSources_1125898_23.stderr parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 5ccb81437294140907d42a8d414dd1e3 parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/596e536d-c1c2-42ea-b9b5-f3ba2a0c6bcf_isrForCrosstalkSources_1125934_52.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/596e536d-c1c2-42ea-b9b5-f3ba2a0c6bcf_isrForCrosstalkSources_1125934_52.stderr parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 7f45f658200181742d42c8c2b52e9e2c parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/993ff383-a64a-4851-aa19-f04c833a6088_isrForCrosstalkSources_1125894_5.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/993ff383-a64a-4851-aa19-f04c833a6088_isrForCrosstalkSources_1125894_5.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash 90a9db2bc7c8c615b0d12d0cbe884b34 parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/91d54c1b-d76e-484e-821e-b2d525e340a2_isrForCrosstalkSources_1125909_42.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/91d54c1b-d76e-484e-821e-b2d525e340a2_isrForCrosstalkSources_1125909_42.stderr parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash 95ea6465f85a4e2b7c9dfb8329cd2c3b parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9967a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/a6a74319-934a-4195-8460-d3b63e157669_isrForCrosstalkSources_1125911_35.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/a6a74319-934a-4195-8460-d3b63e157669_isrForCrosstalkSources_1125911_35.stderr parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 9ea5497f1957478c66eef222bbc5a0fd parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9968e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/b2008e7f-4914-44dc-a889-9a8c9a1d1c31_isrForCrosstalkSources_1125912_44.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/b2008e7f-4914-44dc-a889-9a8c9a1d1c31_isrForCrosstalkSources_1125912_44.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 0629041c009f02c39b6cd258729a8d8c parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/ccc607d2-ae46-4e18-991d-3eeff7b4324d_isrForCrosstalkSources_1125911_30.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/ccc607d2-ae46-4e18-991d-3eeff7b4324d_isrForCrosstalkSources_1125911_30.stderr parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash efea5190451f6747f66d541c7128fd3e parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/233f97a1-3fd9-4c2a-8ce0-0491d52e01e9_isrForCrosstalkSources_1125907_46.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/233f97a1-3fd9-4c2a-8ce0-0491d52e01e9_isrForCrosstalkSources_1125907_46.stderr parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash 4cc4d9e4a81ebbe4c1af358a7f638d7f parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/247b2108-9e82-4689-a91f-5fbae134aef1_isrForCrosstalkSources_1125913_6.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/247b2108-9e82-4689-a91f-5fbae134aef1_isrForCrosstalkSources_1125913_6.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash 3cc23312bdd7b359830e024acedf4a2e parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/0c8965b5-de4e-4444-9fcf-4160ee0ee120_isrForCrosstalkSources_1125941_17.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/0c8965b5-de4e-4444-9fcf-4160ee0ee120_isrForCrosstalkSources_1125941_17.stderr parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash d4bf4e62e260beebbaa36e34c9e0321a parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3c46b709-60bd-4ba7-b5a6-080074f425fb_isrForCrosstalkSources_1125920_55.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3c46b709-60bd-4ba7-b5a6-080074f425fb_isrForCrosstalkSources_1125920_55.stderr parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash 681fa744e474781d7d891e9957dee28a parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/5ed561d1-16a1-4464-974b-46551fc0200e_isrForCrosstalkSources_1125890_59.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/5ed561d1-16a1-4464-974b-46551fc0200e_isrForCrosstalkSources_1125890_59.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash 1fbdb6c2b1c845930fed3ec36adb5caa parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/1acc40dc-783f-4499-a5f4-55ed89127e7a_isrForCrosstalkSources_1125940_44.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/1acc40dc-783f-4499-a5f4-55ed89127e7a_isrForCrosstalkSources_1125940_44.stderr parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash 4e0b98964f4e15c5c3ec23a00a07eac8 parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/5a184a88-f925-43ea-9324-beda720d23e7_isrForCrosstalkSources_1125910_9.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/5a184a88-f925-43ea-9324-beda720d23e7_isrForCrosstalkSources_1125910_9.stderr parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash 4432d6e172d980635bd55b852712d47e parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/2eb00bba-8ac0-42a6-9c81-00ce2e82ae15_isrForCrosstalkSources_1125925_18.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/2eb00bba-8ac0-42a6-9c81-00ce2e82ae15_isrForCrosstalkSources_1125925_18.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 622abfcf8ea336c472a9492a3d5a91f3 parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/5d6f4aaf-d480-4c74-9022-bb8da7e0bfa1_isrForCrosstalkSources_1125897_36.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/5d6f4aaf-d480-4c74-9022-bb8da7e0bfa1_isrForCrosstalkSources_1125897_36.stderr parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash e176b01694c90dbac99f0cceccd38777 parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c996fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/7e25dec5-3800-42ce-987a-3da04a962cca_isrForCrosstalkSources_1125893_5.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/7e25dec5-3800-42ce-987a-3da04a962cca_isrForCrosstalkSources_1125893_5.stderr parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash 07e41be4256276543b297daa93ee84a0 parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/e337be1e-7f1f-4701-9776-2d571c2b6b19_isrForCrosstalkSources_1125926_32.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/e337be1e-7f1f-4701-9776-2d571c2b6b19_isrForCrosstalkSources_1125926_32.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash c613ad7855c7c2d3efeddafeb62c2660 parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9971a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/225a948a-3067-49e0-984b-0d26c20ab451_isrForCrosstalkSources_1125930_40.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/225a948a-3067-49e0-984b-0d26c20ab451_isrForCrosstalkSources_1125930_40.stderr parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash 500d884f428c06316087f005a0672d8d parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/df66fa8a-7f6f-4394-9560-ddd43ba24685_isrForCrosstalkSources_1125908_41.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/df66fa8a-7f6f-4394-9560-ddd43ba24685_isrForCrosstalkSources_1125908_41.stderr parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash 86847d5139f611c39f543f7dfc9c9a0f parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/6f12ef8d-3dfe-4319-b323-1c737da2aa78_isrForCrosstalkSources_1125896_9.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/6f12ef8d-3dfe-4319-b323-1c737da2aa78_isrForCrosstalkSources_1125896_9.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash c5a83e405b7b538d6c59748334a63a24 parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9972e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/a7b22a2c-4caf-4efa-8c7f-54c20eed8b66_isrForCrosstalkSources_1125922_46.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/a7b22a2c-4caf-4efa-8c7f-54c20eed8b66_isrForCrosstalkSources_1125922_46.stderr parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash 99a7b207fa99d8dbd53baeb1e540ae2a parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8aa2aa51-6585-4a0e-b62e-c534dfec92cc_isrForCrosstalkSources_1125936_14.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8aa2aa51-6585-4a0e-b62e-c534dfec92cc_isrForCrosstalkSources_1125936_14.stderr parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash 5da51584d2cb8204b6f6811bfdc94f39 parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9974c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/031cbf43-48ae-482b-8ff2-15c91296420a_isrForCrosstalkSources_1125894_21.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/031cbf43-48ae-482b-8ff2-15c91296420a_isrForCrosstalkSources_1125894_21.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 92c5525b157a8aae7e159c399d449c1b parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/ac9b54f3-7b67-4529-9baf-e85b912d99f1_isrForCrosstalkSources_1125925_42.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/ac9b54f3-7b67-4529-9baf-e85b912d99f1_isrForCrosstalkSources_1125925_42.stderr parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash ef5e62b8945404d9e46c0ca236c2be24 parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/6353627f-4d18-4228-a110-8fe2d71dd20c_isrForCrosstalkSources_1125919_25.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/6353627f-4d18-4228-a110-8fe2d71dd20c_isrForCrosstalkSources_1125919_25.stderr parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash ec97fb575cf705bda5ac12fd79b64a72 parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/46681dde-bcba-4b84-8383-6c2149542737_isrForCrosstalkSources_1125946_19.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/46681dde-bcba-4b84-8383-6c2149542737_isrForCrosstalkSources_1125946_19.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash 29a3b7fac08dffa2799de8dc0a514eb9 parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9976a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/4514723f-98f8-4c72-b670-086bb897dfc8_isrForCrosstalkSources_1125898_26.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/4514723f-98f8-4c72-b670-086bb897dfc8_isrForCrosstalkSources_1125898_26.stderr parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash 6cfb8f815c2ca0ee4ce96f7dc90427db parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9977e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/bc4b82f7-bb06-4758-9ce2-da7986d6487b_isrForCrosstalkSources_1125900_18.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/bc4b82f7-bb06-4758-9ce2-da7986d6487b_isrForCrosstalkSources_1125900_18.stderr parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash 422bf65398209aff8474dae6b9b2c6a0 parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/c26b224e-d616-4d0d-86cb-968c0a291ad4_isrForCrosstalkSources_1125924_32.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/c26b224e-d616-4d0d-86cb-968c0a291ad4_isrForCrosstalkSources_1125924_32.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash 971aa4317d1e160cbd6d3831a5890719 parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/ec1e1eff-0d9a-4849-bad4-68a9b5aa60ec_isrForCrosstalkSources_1125932_30.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/ec1e1eff-0d9a-4849-bad4-68a9b5aa60ec_isrForCrosstalkSources_1125932_30.stderr parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash aeb6696f2a06874e177f0a2d698abd0a parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/d46cc4ef-6231-4567-b4f8-dfe5aabff808_isrForCrosstalkSources_1125902_52.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/d46cc4ef-6231-4567-b4f8-dfe5aabff808_isrForCrosstalkSources_1125902_52.stderr parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash 592ea45dce2624b6e60d9e68f2547988 parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c9979c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/5521f06d-786b-40a3-af40-340fd7787e4c_isrForCrosstalkSources_1125928_17.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/5521f06d-786b-40a3-af40-340fd7787e4c_isrForCrosstalkSources_1125928_17.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 163efa3e33d13621a4c25a92d5dc682e parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/b0a2cdef-b24c-4d9d-953e-7ba7951dd669_isrForCrosstalkSources_1125945_37.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/b0a2cdef-b24c-4d9d-953e-7ba7951dd669_isrForCrosstalkSources_1125945_37.stderr parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash 367b6d80d7b82c817eb67dae161d4362 parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/81aa76af-4b67-4c18-bf6d-3dd0e10c229e_isrForCrosstalkSources_1125931_34.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/81aa76af-4b67-4c18-bf6d-3dd0e10c229e_isrForCrosstalkSources_1125931_34.stderr parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash d9c4d59204a30970521b4d0c2b065460 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/36b54691-a54a-4038-8e39-d19d8a0fe1b3_isrForCrosstalkSources_1125910_35.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/36b54691-a54a-4038-8e39-d19d8a0fe1b3_isrForCrosstalkSources_1125910_35.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 10724ad20c040ef1770590b3a23ca3ad parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/a6239eef-ff6f-4397-a098-9038a7b9a69e_isrForCrosstalkSources_1125939_5.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/a6239eef-ff6f-4397-a098-9038a7b9a69e_isrForCrosstalkSources_1125939_5.stderr parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash 023027aa52b77f2559a5f2d045d2853f parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/cbe88845-f718-4162-af62-54b2736e383f_isrForCrosstalkSources_1125908_45.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/cbe88845-f718-4162-af62-54b2736e383f_isrForCrosstalkSources_1125908_45.stderr parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash 253feb77f79508eaa938f01c0561c774 parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f5a08d8d-3315-436c-864b-3176031007e6_isrForCrosstalkSources_1125939_27.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f5a08d8d-3315-436c-864b-3176031007e6_isrForCrosstalkSources_1125939_27.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash 35a8fa5737a398b03a8f647d704043fa parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/366d268b-0d8c-4f4e-8b2c-70cd4a6f4909_isrForCrosstalkSources_1125919_14.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/366d268b-0d8c-4f4e-8b2c-70cd4a6f4909_isrForCrosstalkSources_1125919_14.stderr parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash 012117b1c202b8cec3f4ab5511cc197b parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/63361b7a-fe3d-4a9e-90c5-579f61f36e7f_isrForCrosstalkSources_1125901_53.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/63361b7a-fe3d-4a9e-90c5-579f61f36e7f_isrForCrosstalkSources_1125901_53.stderr parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash feb0553034b3766a9a7adb9aae347950 parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c997f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/20aa09ba-b0d1-4493-a05f-7b81df107ee9_isrForCrosstalkSources_1125941_38.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/20aa09ba-b0d1-4493-a05f-7b81df107ee9_isrForCrosstalkSources_1125941_38.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash e8fa3a48a09fe369fcf27392b1abd982 parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3c994040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/a8569b8e-40fd-4896-8821-fa224d46deb2_isrForCrosstalkSources_1125927_21.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/a8569b8e-40fd-4896-8821-fa224d46deb2_isrForCrosstalkSources_1125927_21.stderr parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash 34e0e5fb0bc94132ac32cabe31bece18 parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/8b611a6b-292a-43cc-8418-bcbd4bc08511_isrForCrosstalkSources_1125889_32.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/8b611a6b-292a-43cc-8418-bcbd4bc08511_isrForCrosstalkSources_1125889_32.stderr parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash 8bbb1bbddfd28f12c67a0b902cd1ff42 parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/6e25d469-3333-4ea2-82dd-c8e65bebee68_isrForCrosstalkSources_1125913_19.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/6e25d469-3333-4ea2-82dd-c8e65bebee68_isrForCrosstalkSources_1125913_19.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash 72df495729545fa270e847dc7ee2501d parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/a2a11641-d8d4-42a2-8547-60f78a81ae5d_isrForCrosstalkSources_1125898_41.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/a2a11641-d8d4-42a2-8547-60f78a81ae5d_isrForCrosstalkSources_1125898_41.stderr parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash eba169faf11c4df5feff5b29b73ee85e parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/1c3f07c1-fad2-42a1-8e7a-83e6cb6d4e87_isrForCrosstalkSources_1125912_9.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/1c3f07c1-fad2-42a1-8e7a-83e6cb6d4e87_isrForCrosstalkSources_1125912_9.stderr parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash 1e783a7a41569e43dffaa4006c4eb8aa parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/e9a53ee4-38ff-42ff-ba2e-5c2b57846b65_isrForCrosstalkSources_1125912_36.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/e9a53ee4-38ff-42ff-ba2e-5c2b57846b65_isrForCrosstalkSources_1125912_36.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash 1c588bafbc93932f744ea6e61fe674aa parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/539a2542-721a-41e3-a345-4eceaa73b46c_isrForCrosstalkSources_1125918_47.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/539a2542-721a-41e3-a345-4eceaa73b46c_isrForCrosstalkSources_1125918_47.stderr parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 13757561842a481f8d27fcc51d775225 parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/40b394fa-9d01-4272-9f1e-d5e4be7befff_isrForCrosstalkSources_1125913_40.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/40b394fa-9d01-4272-9f1e-d5e4be7befff_isrForCrosstalkSources_1125913_40.stderr parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash 0b70202c0cd6edb857cb1dbd512623cd parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/621734b9-9c34-41b5-bbd2-c0650bfc15df_isrForCrosstalkSources_1125934_31.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/621734b9-9c34-41b5-bbd2-c0650bfc15df_isrForCrosstalkSources_1125934_31.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 70e581e55f848aa8e74a0487e7e9700d parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/31a2be85-c702-4827-ad13-4babecafc25c_isrForCrosstalkSources_1125889_52.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/31a2be85-c702-4827-ad13-4babecafc25c_isrForCrosstalkSources_1125889_52.stderr parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash b18d792643147d3ffc39dcc201d6e1a3 parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4ac203ed-bfef-45cf-9e24-fcfa7e785518_isrForCrosstalkSources_1125893_50.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4ac203ed-bfef-45cf-9e24-fcfa7e785518_isrForCrosstalkSources_1125893_50.stderr parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash 732a81405d34715da3741c9c5faa743e parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/3ef2618e-1ac0-490e-a402-aef4eb1f9b28_isrForCrosstalkSources_1125944_33.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/3ef2618e-1ac0-490e-a402-aef4eb1f9b28_isrForCrosstalkSources_1125944_33.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash 8c5372ad2b2d6c2d768e3dd0bcdd407b parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/eaf52586-378c-45b4-a4aa-5e8540cff5cf_isrForCrosstalkSources_1125930_60.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/eaf52586-378c-45b4-a4aa-5e8540cff5cf_isrForCrosstalkSources_1125930_60.stderr parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash 7256a2b2a0cdeb8e03212cd0786ad590 parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/a4ece2b7-bf06-424d-8a17-c607e7e6f11a_isrForCrosstalkSources_1125930_11.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/a4ece2b7-bf06-424d-8a17-c607e7e6f11a_isrForCrosstalkSources_1125930_11.stderr parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash 3f8cac71ca91a8eb1bcee62c25640fb6 parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/95c27012-fb59-4652-a06e-229b77d230f7_isrForCrosstalkSources_1125905_44.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/95c27012-fb59-4652-a06e-229b77d230f7_isrForCrosstalkSources_1125905_44.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash 4eee2bd9b081800c0a814f130e71a4fe parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/21abffef-4bca-4ef1-ae64-55ec64956c1f_isrForCrosstalkSources_1125891_44.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/21abffef-4bca-4ef1-ae64-55ec64956c1f_isrForCrosstalkSources_1125891_44.stderr parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash 27ef68946ed1f38e45142781aa1c276a parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f8d1e1b4-b675-4242-a931-4d9c09e09161_isrForCrosstalkSources_1125897_17.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f8d1e1b4-b675-4242-a931-4d9c09e09161_isrForCrosstalkSources_1125897_17.stderr parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash b8a7b21a74123b1589b535b733bf10f4 parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/301c12ec-5d96-4f98-a068-6904607766e5_isrForCrosstalkSources_1125918_4.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/301c12ec-5d96-4f98-a068-6904607766e5_isrForCrosstalkSources_1125918_4.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash 2aa3e3fc00947367030945343c438a2d parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/022294a5-1fc4-49d3-adf6-8290fa3b1519_isrForCrosstalkSources_1125894_49.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/022294a5-1fc4-49d3-adf6-8290fa3b1519_isrForCrosstalkSources_1125894_49.stderr parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash 002d5f6239081b426291a28fdd31ce22 parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/14c6834f-fb61-4647-a7c6-1924161f1176_isrForCrosstalkSources_1125902_21.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/14c6834f-fb61-4647-a7c6-1924161f1176_isrForCrosstalkSources_1125902_21.stderr parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash 79f1a919ed832fa5a68aa29e59dcca48 parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/04ec693a-ce42-4882-b4cd-7c10d2603fa9_isrForCrosstalkSources_1125930_13.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/04ec693a-ce42-4882-b4cd-7c10d2603fa9_isrForCrosstalkSources_1125930_13.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash 9bbc7725d4c91f267d63169256fe7382 parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/1c42f905-753b-4d97-93e5-9b67c26ff6fd_isrForCrosstalkSources_1125923_25.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/1c42f905-753b-4d97-93e5-9b67c26ff6fd_isrForCrosstalkSources_1125923_25.stderr parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash b4c1ca0ce1fa2ba036d6589e6255a8de parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/f7e4699c-be4b-40ee-bb80-8dad47c35c0e_isrForCrosstalkSources_1125909_40.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/f7e4699c-be4b-40ee-bb80-8dad47c35c0e_isrForCrosstalkSources_1125909_40.stderr parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash 6c0f8e9bbb3fb6f8746537e96dd13472 parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/93e8928c-485d-4231-b778-2cea8b62b20f_isrForCrosstalkSources_1125915_54.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/93e8928c-485d-4231-b778-2cea8b62b20f_isrForCrosstalkSources_1125915_54.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash 1d42085be80e6c8101b24201a3619551 parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/6da12762-6447-48fe-a5c8-2dbd19077bb3_isrForCrosstalkSources_1125907_34.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/6da12762-6447-48fe-a5c8-2dbd19077bb3_isrForCrosstalkSources_1125907_34.stderr parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash b9f43e800ab83569b3f7c578df75022b parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c25acaaa-588b-4a4a-aab9-ce49e416513f_isrForCrosstalkSources_1125893_21.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c25acaaa-588b-4a4a-aab9-ce49e416513f_isrForCrosstalkSources_1125893_21.stderr parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash f3491a4a9e5d7089878090c4a98d2ad7 parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/41750dd5-33bb-4aa8-b1e8-df7901670e70_isrForCrosstalkSources_1125930_35.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/41750dd5-33bb-4aa8-b1e8-df7901670e70_isrForCrosstalkSources_1125930_35.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 789c7a10283d9efb588c2cfa640f28a7 parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/1ba0546e-f382-491f-a897-5217be68fac1_isrForCrosstalkSources_1125921_40.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/1ba0546e-f382-491f-a897-5217be68fac1_isrForCrosstalkSources_1125921_40.stderr parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash 241437833f28c72a2f26957df12c8828 parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/f14312a7-fd98-4db2-a286-005ddf68d5f6_isrForCrosstalkSources_1125923_9.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/f14312a7-fd98-4db2-a286-005ddf68d5f6_isrForCrosstalkSources_1125923_9.stderr parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash 62195175ab797a505a83db35afad38d0 parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/6e720da9-f615-4ceb-a8cf-034dccd25c52_isrForCrosstalkSources_1125891_52.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/6e720da9-f615-4ceb-a8cf-034dccd25c52_isrForCrosstalkSources_1125891_52.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash 6810e934b0156d71e41208cd635a0aa4 parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d1aa5469-3487-4e6c-aa75-650536bd1037_isrForCrosstalkSources_1125915_41.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d1aa5469-3487-4e6c-aa75-650536bd1037_isrForCrosstalkSources_1125915_41.stderr parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash 3096d09d7728b442833578038ac95d4c parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/47c80003-b1dc-4290-8d52-84a9a3e4cf3d_isrForCrosstalkSources_1125898_61.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/47c80003-b1dc-4290-8d52-84a9a3e4cf3d_isrForCrosstalkSources_1125898_61.stderr parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash afb945571529459c90bba5e42ccca65e parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/5119c7dc-cb34-4e0b-924b-2f65f7387369_isrForCrosstalkSources_1125923_59.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/5119c7dc-cb34-4e0b-924b-2f65f7387369_isrForCrosstalkSources_1125923_59.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash b4d849158bf3cc80bb168a4ff1f84abe parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/fcf73a54-5016-4c69-b252-ac8bedf7e052_isrForCrosstalkSources_1125922_37.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/fcf73a54-5016-4c69-b252-ac8bedf7e052_isrForCrosstalkSources_1125922_37.stderr parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 8d4d5dac6c116efa61da7319b0dd5f2b parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/28bf4a89-9ae6-4c92-96e8-1ed4ffba3f43_isrForCrosstalkSources_1125896_60.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/28bf4a89-9ae6-4c92-96e8-1ed4ffba3f43_isrForCrosstalkSources_1125896_60.stderr parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash f8f88f62df28a26a5802810e3e110683 parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/8759f061-09bc-493b-b0db-67f79d7fd9e8_isrForCrosstalkSources_1125893_24.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/8759f061-09bc-493b-b0db-67f79d7fd9e8_isrForCrosstalkSources_1125893_24.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash 9aaf989161e1343c64428cdf6c53a772 parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/495a7a3c-12e0-4768-a998-f010602dc891_isrForCrosstalkSources_1125907_39.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/495a7a3c-12e0-4768-a998-f010602dc891_isrForCrosstalkSources_1125907_39.stderr parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 98797972456685f00f7ed859b3ea0025 parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/ebf9d239-f0e9-484e-9ddd-4d570461a13e_isrForCrosstalkSources_1125913_11.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/ebf9d239-f0e9-484e-9ddd-4d570461a13e_isrForCrosstalkSources_1125913_11.stderr parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash dd9405ddd79dedac2b7914362d299fb5 parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/56058fdd-2950-4664-b6f0-68ecf0de90b4_isrForCrosstalkSources_1125915_3.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/56058fdd-2950-4664-b6f0-68ecf0de90b4_isrForCrosstalkSources_1125915_3.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash 6c5c3bd9f8d9412109479a37d286d500 parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/02cf7d1e-d661-413c-81f0-d84c24927a0c_isrForCrosstalkSources_1125893_36.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/02cf7d1e-d661-413c-81f0-d84c24927a0c_isrForCrosstalkSources_1125893_36.stderr parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash a440afbd725e5ea9af97339e3e3be260 parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2d37a09b-8b5a-4753-a28a-9f91df1bc949_isrForCrosstalkSources_1125895_62.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2d37a09b-8b5a-4753-a28a-9f91df1bc949_isrForCrosstalkSources_1125895_62.stderr parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash a4ae44f5bdac31f8bed9bb9282f91e8c parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/140176d7-df21-4a37-a608-5d0975c9b234_isrForCrosstalkSources_1125900_49.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/140176d7-df21-4a37-a608-5d0975c9b234_isrForCrosstalkSources_1125900_49.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash d92d62b1158fa4d064d71feae19275aa parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/90852f11-d676-474d-be6b-7079e60ec965_isrForCrosstalkSources_1125895_54.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/90852f11-d676-474d-be6b-7079e60ec965_isrForCrosstalkSources_1125895_54.stderr parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash f9b788829789d168d04af09ef35954c5 parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/25b2a31c-7ee8-4280-8bbd-96a4a307cfea_isrForCrosstalkSources_1125932_53.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/25b2a31c-7ee8-4280-8bbd-96a4a307cfea_isrForCrosstalkSources_1125932_53.stderr parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash 1aee264e802e1493d66fbcc0e9d559f6 parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/6602bd34-e0b7-4211-8c6f-1a31093a7352_isrForCrosstalkSources_1125930_3.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/6602bd34-e0b7-4211-8c6f-1a31093a7352_isrForCrosstalkSources_1125930_3.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash c5dc8b15253b2aa777486cd80d4696d5 parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/58910a69-8b62-4c88-a54e-03c6014b3f9f_isrForCrosstalkSources_1125906_32.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/58910a69-8b62-4c88-a54e-03c6014b3f9f_isrForCrosstalkSources_1125906_32.stderr parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash 468f77d37cb2cfff7b84d4b435ab8dcc parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/e73fa079-f05f-484b-8c18-5d01338de168_isrForCrosstalkSources_1125891_15.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/e73fa079-f05f-484b-8c18-5d01338de168_isrForCrosstalkSources_1125891_15.stderr parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 514846a73e1b7b89435ffc90b0eb252a parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/22a55563-7821-49ae-b04a-bad6ebe29680_isrForCrosstalkSources_1125906_21.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/22a55563-7821-49ae-b04a-bad6ebe29680_isrForCrosstalkSources_1125906_21.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash 060d76acb69e8139f1976f3d362eb689 parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/82efe159-dfbf-4b53-a4e5-ed67790a4945_isrForCrosstalkSources_1125891_57.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/82efe159-dfbf-4b53-a4e5-ed67790a4945_isrForCrosstalkSources_1125891_57.stderr parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash 1f1751af7d22817dfc7e12da2416d4d8 parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/06bc21c6-5dab-4c1b-9ec3-8739e4d70325_isrForCrosstalkSources_1125946_46.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/06bc21c6-5dab-4c1b-9ec3-8739e4d70325_isrForCrosstalkSources_1125946_46.stderr parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash c4413f237b4727604c19f82f7659d945 parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/26588652-1351-4faf-a049-84ce0a2c267e_isrForCrosstalkSources_1125905_7.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/26588652-1351-4faf-a049-84ce0a2c267e_isrForCrosstalkSources_1125905_7.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash 3519e17395ca9b4200441fa889a9e506 parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/7b98f1bc-2068-47d4-a7ac-11a19dc88542_isrForCrosstalkSources_1125906_53.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/7b98f1bc-2068-47d4-a7ac-11a19dc88542_isrForCrosstalkSources_1125906_53.stderr parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash f2364fbd0e62855b17bcadf4adb8761c parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/66bcbcbf-0154-4651-bfff-00bff8af3161_isrForCrosstalkSources_1125909_24.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/66bcbcbf-0154-4651-bfff-00bff8af3161_isrForCrosstalkSources_1125909_24.stderr parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash 3461108ff6f678392c0d3795ee221fdb parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/c5b58a16-bdf3-4b5c-a129-c95f015348f2_isrForCrosstalkSources_1125926_38.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/c5b58a16-bdf3-4b5c-a129-c95f015348f2_isrForCrosstalkSources_1125926_38.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash cb4e0fe894513631cbecb2326b0dd054 parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/a5dbaa89-a33c-4697-b881-358a66a52f23_isrForCrosstalkSources_1125929_3.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/a5dbaa89-a33c-4697-b881-358a66a52f23_isrForCrosstalkSources_1125929_3.stderr parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash c983db0a1b1c7ad299bd2aea988675a6 parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/29f6df08-e30b-48c6-a637-7cab746d7dcf_isrForCrosstalkSources_1125932_38.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/29f6df08-e30b-48c6-a637-7cab746d7dcf_isrForCrosstalkSources_1125932_38.stderr parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash 136d1646d10ecb073d68c0f8dffe7f45 parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b0ed425e-1a86-4d49-8f9d-04deef0d1998_isrForCrosstalkSources_1125934_14.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b0ed425e-1a86-4d49-8f9d-04deef0d1998_isrForCrosstalkSources_1125934_14.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash 28079513379b43018bda2925a03bc5a5 parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/13d4dab4-1995-4a1a-891c-c218fe60133b_isrForCrosstalkSources_1125924_43.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/13d4dab4-1995-4a1a-891c-c218fe60133b_isrForCrosstalkSources_1125924_43.stderr parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash 6127bb6b03ebf71d750669cd5badd931 parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/a143b94a-b461-45bc-81b6-ddd26ba0d609_isrForCrosstalkSources_1125911_57.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/a143b94a-b461-45bc-81b6-ddd26ba0d609_isrForCrosstalkSources_1125911_57.stderr parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash 3a84e663b09c2ed2e0ff84d6f6f7fc69 parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/35526065-bd2d-4211-a6e3-bfbba08d137d_isrForCrosstalkSources_1125930_50.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/35526065-bd2d-4211-a6e3-bfbba08d137d_isrForCrosstalkSources_1125930_50.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 011465ade036ddbf2b65b61bb51d15fe parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:40 proc_lsst.multi:146 [INFO] found job 53087 in provider local proc_lsst.multi INFO: found job 53087 in provider local parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/6cc12926-b68d-4189-a34a-3360d2ed97bb_isrForCrosstalkSources_1125889_27.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/6cc12926-b68d-4189-a34a-3360d2ed97bb_isrForCrosstalkSources_1125889_27.stderr parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash 054c7b9479aef1ccb5f8aee602c78ab2 parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/f64c1663-3b81-44c0-b833-79ec598b3de3_isrForCrosstalkSources_1125915_21.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/f64c1663-3b81-44c0-b833-79ec598b3de3_isrForCrosstalkSources_1125915_21.stderr parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash 1b7873d2739a1e95e8d3ef81736fa273 parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/349a0efd-3836-4a73-b2f5-124c0421f07d_isrForCrosstalkSources_1125940_22.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/349a0efd-3836-4a73-b2f5-124c0421f07d_isrForCrosstalkSources_1125940_22.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash 7a0186b98dd89e3c41ebffcccb817bca parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/086d78e5-1e60-4692-a774-0933cd388f91_isrForCrosstalkSources_1125896_2.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/086d78e5-1e60-4692-a774-0933cd388f91_isrForCrosstalkSources_1125896_2.stderr parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash c38675da7d7478dd0e04fd3431fe1762 parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c1732c9f-5bff-45c7-a934-b110bf4e75bc_isrForCrosstalkSources_1125910_15.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c1732c9f-5bff-45c7-a934-b110bf4e75bc_isrForCrosstalkSources_1125910_15.stderr parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash 148a9721bd49d277156997f0a30e4c22 parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/a2e4a30f-f43b-4b58-b3f1-686875e1893d_isrForCrosstalkSources_1125926_51.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/a2e4a30f-f43b-4b58-b3f1-686875e1893d_isrForCrosstalkSources_1125926_51.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash 7d661c60e4cf8800b5d2620d3da0d8bc parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 487 parsl.dataflow.strategy DEBUG: Executor multi has 487 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 121 more blocks parsl.executors.status_handling INFO: Scaling out by 121 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-07 15:06:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-07 15:06:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 15:06:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/feed14e1-bd72-4de0-98e6-04a9663574e9_isrForCrosstalkSources_1125938_46.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/feed14e1-bd72-4de0-98e6-04a9663574e9_isrForCrosstalkSources_1125938_46.stderr parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash 3c894b118778f2d1c4cc7899c6dfa024 parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/5ed94791-e096-4438-813d-a6f1cebde254_isrForCrosstalkSources_1125918_62.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/5ed94791-e096-4438-813d-a6f1cebde254_isrForCrosstalkSources_1125918_62.stderr parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash 4c834cb02b05317f7e2e5fc2a70bf726 parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/88cac695-4245-4493-bf37-34650be59f71_isrForCrosstalkSources_1125921_18.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/88cac695-4245-4493-bf37-34650be59f71_isrForCrosstalkSources_1125921_18.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash 018b9429b39a0279de18d3f14c368824 parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/718054d1-03e9-4e4e-9a4d-2e99f699d1a1_isrForCrosstalkSources_1125896_46.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/718054d1-03e9-4e4e-9a4d-2e99f699d1a1_isrForCrosstalkSources_1125896_46.stderr parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash 663de4db32ef7369ac0c195ef5f079b6 parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/f64ba87d-eeea-46fa-a0df-0ebc937b2c22_isrForCrosstalkSources_1125891_54.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/f64ba87d-eeea-46fa-a0df-0ebc937b2c22_isrForCrosstalkSources_1125891_54.stderr parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash d90f6ad0354284e9d49bebb41577659b parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/17809f5b-d724-4a2c-9af7-afd3d21952cb_isrForCrosstalkSources_1125892_6.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/17809f5b-d724-4a2c-9af7-afd3d21952cb_isrForCrosstalkSources_1125892_6.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash bccb0148c5a80772ea99cd88ea0beea0 parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/d68be105-faed-4fa6-af3e-7aa9b1a086cb_isrForCrosstalkSources_1125901_43.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/d68be105-faed-4fa6-af3e-7aa9b1a086cb_isrForCrosstalkSources_1125901_43.stderr parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash 177cec6fa1250047656055b72ee99333 parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/e938b9d1-e2da-4713-849c-124892214b63_isrForCrosstalkSources_1125927_9.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/e938b9d1-e2da-4713-849c-124892214b63_isrForCrosstalkSources_1125927_9.stderr parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash a8b615797c622361be9be8e1f56ec5a6 parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/e546d399-6ef4-4440-8f99-73f7d3770200_isrForCrosstalkSources_1125905_6.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/e546d399-6ef4-4440-8f99-73f7d3770200_isrForCrosstalkSources_1125905_6.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash 7b1776cf9e4f6c1b79d97b6922ca3bcb parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ba3bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3372171c-c045-4389-878b-f09f29eb27e6_isrForCrosstalkSources_1125941_47.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3372171c-c045-4389-878b-f09f29eb27e6_isrForCrosstalkSources_1125941_47.stderr parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash a253c57d9e73f7cccab123acb1d30095 parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/54d7c0de-4684-4305-8008-97fa6db7ec01_isrForCrosstalkSources_1125899_11.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/54d7c0de-4684-4305-8008-97fa6db7ec01_isrForCrosstalkSources_1125899_11.stderr parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash d5bb952f32d5c5de6fa44c3a6acf9be1 parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/f8dba169-4a64-4351-a8c2-9e218fe90978_isrForCrosstalkSources_1125903_18.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/f8dba169-4a64-4351-a8c2-9e218fe90978_isrForCrosstalkSources_1125903_18.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash 26e00e2fa255a2882a6c0394c2d7ce5d parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/0db15fde-958f-4372-8ac6-d9759ee3a8ae_isrForCrosstalkSources_1125927_11.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/0db15fde-958f-4372-8ac6-d9759ee3a8ae_isrForCrosstalkSources_1125927_11.stderr parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash 96deba7b967ac76670ae2236e769fb50 parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/377ef147-0129-4a72-a4ec-ba990d45d0a3_isrForCrosstalkSources_1125894_41.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/377ef147-0129-4a72-a4ec-ba990d45d0a3_isrForCrosstalkSources_1125894_41.stderr parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash 2e1aafce9550198b64fbf6376ae5e451 parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/2826457e-d018-49ee-9887-e349163714c8_isrForCrosstalkSources_1125923_19.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/2826457e-d018-49ee-9887-e349163714c8_isrForCrosstalkSources_1125923_19.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash 288a89b251aa4322ca8328a1abf79753 parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/3ae108cb-e837-4258-ad7e-701c068e3172_isrForCrosstalkSources_1125937_16.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/3ae108cb-e837-4258-ad7e-701c068e3172_isrForCrosstalkSources_1125937_16.stderr parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash dcca4c993ae01778e42067007a09a884 parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4fe26ac8-87b8-4a2c-8f41-cac6578ef2cf_isrForCrosstalkSources_1125908_26.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4fe26ac8-87b8-4a2c-8f41-cac6578ef2cf_isrForCrosstalkSources_1125908_26.stderr parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash 0a7ed80bdf719a9c8ea4fd0438417acd parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/196e2add-df0b-4b40-81b0-87e76d4c6a40_isrForCrosstalkSources_1125945_11.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/196e2add-df0b-4b40-81b0-87e76d4c6a40_isrForCrosstalkSources_1125945_11.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash 4bbb19dea926c7c6c62a5af107b9bb0e parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/9aa77323-411d-4fd5-9d7e-8a94b28bc577_isrForCrosstalkSources_1125911_52.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/9aa77323-411d-4fd5-9d7e-8a94b28bc577_isrForCrosstalkSources_1125911_52.stderr parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash c8a25c5c4c7601d2a485a48222aab9c3 parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/e3d7dfb0-5cbc-413e-9e7e-166997869a1f_isrForCrosstalkSources_1125925_35.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/e3d7dfb0-5cbc-413e-9e7e-166997869a1f_isrForCrosstalkSources_1125925_35.stderr parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash 63a5005ef1e73349afcafb7d70534817 parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/f929ad59-51cd-4441-85b4-d82006aa0206_isrForCrosstalkSources_1125931_39.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/f929ad59-51cd-4441-85b4-d82006aa0206_isrForCrosstalkSources_1125931_39.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 3c3be0b8f278013192b5a9631eedc745 parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/80c826ed-2f80-4279-a5fe-10047729b064_isrForCrosstalkSources_1125930_5.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/80c826ed-2f80-4279-a5fe-10047729b064_isrForCrosstalkSources_1125930_5.stderr parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash ff680d63bff55fc95510edd567451c59 parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/5e6ccfa9-b52f-4fdb-b2cf-38fd9137455b_isrForCrosstalkSources_1125939_43.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/5e6ccfa9-b52f-4fdb-b2cf-38fd9137455b_isrForCrosstalkSources_1125939_43.stderr parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 3dca0ac3f3983882fb2e250d9f6a2bed parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/9e21793a-66d8-4eb4-bdae-6b9836608e87_isrForCrosstalkSources_1125942_39.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/9e21793a-66d8-4eb4-bdae-6b9836608e87_isrForCrosstalkSources_1125942_39.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash d6993791a509e18f5e962da2611859d3 parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/3d72ae97-02cc-4126-b263-54076e553b12_isrForCrosstalkSources_1125924_38.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/3d72ae97-02cc-4126-b263-54076e553b12_isrForCrosstalkSources_1125924_38.stderr parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash 915ce6c73b387df71e04dc9a7acf7f62 parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/abd322ea-9e2d-4e07-877d-01de1989b3f0_isrForCrosstalkSources_1125931_58.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/abd322ea-9e2d-4e07-877d-01de1989b3f0_isrForCrosstalkSources_1125931_58.stderr parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash cc6b28c07f9ea0b570a62afde0c7b518 parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/60d450e1-073c-4ab0-8893-63ada0ee4ebe_isrForCrosstalkSources_1125925_43.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/60d450e1-073c-4ab0-8893-63ada0ee4ebe_isrForCrosstalkSources_1125925_43.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash e3c3a36446b509aa1063c71e92c7de73 parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/df1a5746-c32b-4751-b727-842628cde99c_isrForCrosstalkSources_1125888_43.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/df1a5746-c32b-4751-b727-842628cde99c_isrForCrosstalkSources_1125888_43.stderr parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash 01699d96c219e35e0184ac9134fa0dd0 parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/95723426-7f49-4cb9-9975-5c8f2d26be70_isrForCrosstalkSources_1125905_50.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/95723426-7f49-4cb9-9975-5c8f2d26be70_isrForCrosstalkSources_1125905_50.stderr parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash 4eded6dbc208dac40347e6a9bdd15f6b parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/3c3ffce1-b3d6-4b14-b9f8-26cb5884cbaf_isrForCrosstalkSources_1125918_49.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/3c3ffce1-b3d6-4b14-b9f8-26cb5884cbaf_isrForCrosstalkSources_1125918_49.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash b839fc7a60ac042e4b41a24a4d82a0cd parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/4ec6fbe1-2d2e-4762-a17f-915ffc32a93f_isrForCrosstalkSources_1125888_17.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/4ec6fbe1-2d2e-4762-a17f-915ffc32a93f_isrForCrosstalkSources_1125888_17.stderr parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash 7d1bcc8b7821ad1b5e7c562e94240fb6 parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/74fd42d3-512d-4641-995a-8bf790e9923e_isrForCrosstalkSources_1125933_45.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/74fd42d3-512d-4641-995a-8bf790e9923e_isrForCrosstalkSources_1125933_45.stderr parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash 122a478a560f980a972bbb64e3685299 parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/1017e6ac-feaa-42d4-98b4-473b23dc6dc2_isrForCrosstalkSources_1125896_49.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/1017e6ac-feaa-42d4-98b4-473b23dc6dc2_isrForCrosstalkSources_1125896_49.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash cbc9782e952aeb9175d0be8fbda745c4 parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/25586c3e-46a2-44ab-863e-13ff89307f05_isrForCrosstalkSources_1125917_54.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/25586c3e-46a2-44ab-863e-13ff89307f05_isrForCrosstalkSources_1125917_54.stderr parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash 1375bec2ece52d04fe3b809addc3eff9 parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/d385a75e-1907-4dd4-8d6b-566873299594_isrForCrosstalkSources_1125913_51.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/d385a75e-1907-4dd4-8d6b-566873299594_isrForCrosstalkSources_1125913_51.stderr parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash 270abb54e38b3b5994793db03e4dde56 parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/4c267442-9217-425d-bc85-9715df9e3170_isrForCrosstalkSources_1125888_59.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/4c267442-9217-425d-bc85-9715df9e3170_isrForCrosstalkSources_1125888_59.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash 1bdb7f2f8429e13ac679f710e611abd0 parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/b4273a2c-9e0f-479b-a5c9-14f95a952767_isrForCrosstalkSources_1125936_16.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/b4273a2c-9e0f-479b-a5c9-14f95a952767_isrForCrosstalkSources_1125936_16.stderr parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash ed54fb8e7edebb9f28362ffd3118c0cd parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/bff316f1-1c96-42d4-8d0d-54bee473cd9e_isrForCrosstalkSources_1125897_30.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/bff316f1-1c96-42d4-8d0d-54bee473cd9e_isrForCrosstalkSources_1125897_30.stderr parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash 0c638ccf61927c2640a83023831039e9 parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/1c8c1283-4cbc-4a04-84e7-610707a9b445_isrForCrosstalkSources_1125946_1.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/1c8c1283-4cbc-4a04-84e7-610707a9b445_isrForCrosstalkSources_1125946_1.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash e05556534383ff783c5db208d339d3f8 parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/25b4f6f0-404b-403d-9ecd-ef42aa06a27e_isrForCrosstalkSources_1125926_37.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/25b4f6f0-404b-403d-9ecd-ef42aa06a27e_isrForCrosstalkSources_1125926_37.stderr parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash 706f9ff774ce4bcbdfda7b1a99eb665c parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/df4e23d1-4e46-4a97-a281-482cdab78f81_isrForCrosstalkSources_1125894_58.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/df4e23d1-4e46-4a97-a281-482cdab78f81_isrForCrosstalkSources_1125894_58.stderr parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash 487075abdfefb76d73e3ff397fda1c74 parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:40 proc_lsst.multi:170 [INFO] [multi] job_id 16896385 proc_lsst.multi INFO: [multi] job_id 16896385 2024-03-07 15:06:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896385 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896385 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16896385 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-07 15:06:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-07 15:06:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/4b38c2ec-4d26-4f15-abfb-70e0f2296bb3_isrForCrosstalkSources_1125901_56.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/4b38c2ec-4d26-4f15-abfb-70e0f2296bb3_isrForCrosstalkSources_1125901_56.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash 537c82bf351774f82200d4b4f0c784c9 parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0f4bf9f8-8d71-4f8e-b094-09bb761627fa_isrForCrosstalkSources_1125914_4.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0f4bf9f8-8d71-4f8e-b094-09bb761627fa_isrForCrosstalkSources_1125914_4.stderr parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash d7f6741ebd615dbb5dcd6e6b0a03dc79 parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/51a36469-e70e-4d4f-a04b-43cf629079e9_isrForCrosstalkSources_1125903_34.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/51a36469-e70e-4d4f-a04b-43cf629079e9_isrForCrosstalkSources_1125903_34.stderr parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash d0a1a2cdf0a962617e64667e5ad425eb parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/a85afe2d-24d9-4e23-9c0e-0aa689aef33a_isrForCrosstalkSources_1125911_34.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/a85afe2d-24d9-4e23-9c0e-0aa689aef33a_isrForCrosstalkSources_1125911_34.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 5ceeb9e3a33c257452dd7a6e64ff3ee2 parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/bde58214-4041-403f-b875-1320d69f6259_isrForCrosstalkSources_1125897_54.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/bde58214-4041-403f-b875-1320d69f6259_isrForCrosstalkSources_1125897_54.stderr parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 77c2e2de74f65d6c3ace3acfee9e0271 parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/25544466-6e68-4742-ae9e-e3b6eed92a6d_isrForCrosstalkSources_1125946_23.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/25544466-6e68-4742-ae9e-e3b6eed92a6d_isrForCrosstalkSources_1125946_23.stderr parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash 4e568eeaa0166268f98e04298425b326 parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/b99f1c33-4bda-4f98-a8b9-5115b60bc799_isrForCrosstalkSources_1125912_4.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/b99f1c33-4bda-4f98-a8b9-5115b60bc799_isrForCrosstalkSources_1125912_4.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 31409ad08236ed6f4a2ece67896e7cb3 parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/f393d083-33f3-4b74-b3ed-0b617db0a8aa_isrForCrosstalkSources_1125936_15.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/f393d083-33f3-4b74-b3ed-0b617db0a8aa_isrForCrosstalkSources_1125936_15.stderr parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash c8ffb3d38840b7aa3ca9dde0e9c75af9 parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/446dfe63-a7fa-4a7c-bea3-d95f675e9097_isrForCrosstalkSources_1125888_57.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/446dfe63-a7fa-4a7c-bea3-d95f675e9097_isrForCrosstalkSources_1125888_57.stderr parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash ced9c9df0e57c81d5294a66db783a39d parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/ea61118a-ee15-441a-92a8-5bcf067f237e_isrForCrosstalkSources_1125888_19.stdout parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/ea61118a-ee15-441a-92a8-5bcf067f237e_isrForCrosstalkSources_1125888_19.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 1f8577a7d5377623243cd7d6dd59bec5 parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/ac91cffe-56c0-4882-bc73-7e1a945e404f_isrForCrosstalkSources_1125944_37.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/ac91cffe-56c0-4882-bc73-7e1a945e404f_isrForCrosstalkSources_1125944_37.stderr parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash 5e3974f6fb04ab2af9cbdfcd8665cee4 parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/a25dda5c-fbaf-4bc7-81b5-cd2d2b74262c_isrForCrosstalkSources_1125926_58.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/a25dda5c-fbaf-4bc7-81b5-cd2d2b74262c_isrForCrosstalkSources_1125926_58.stderr parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash 4120b9497e076dff06ed4303965ff421 parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/d1eeaedf-9443-4616-88c2-d533b73753a4_isrForCrosstalkSources_1125926_11.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/d1eeaedf-9443-4616-88c2-d533b73753a4_isrForCrosstalkSources_1125926_11.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash 11be9ae849395291cd6a65ce3e0b1f38 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/2108a32b-7646-4b2a-ab5f-17c1df068b36_isrForCrosstalkSources_1125913_31.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/2108a32b-7646-4b2a-ab5f-17c1df068b36_isrForCrosstalkSources_1125913_31.stderr parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash 3013c838a340863a700b3efe258d42d9 parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/e7f47c40-4932-439d-b5e8-b2308e1a676e_isrForCrosstalkSources_1125943_62.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/e7f47c40-4932-439d-b5e8-b2308e1a676e_isrForCrosstalkSources_1125943_62.stderr parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash 1eaa87eb37dbc9ed0171b6f1269881e2 parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/4ed29e5e-b9b8-41b4-bda6-f462e8a6d6e0_isrForCrosstalkSources_1125900_5.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/4ed29e5e-b9b8-41b4-bda6-f462e8a6d6e0_isrForCrosstalkSources_1125900_5.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash fd7fdbd7ea7f73f40d3e92fb18c33072 parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/d3d28935-b3ad-45f2-8fa9-779473e974d7_isrForCrosstalkSources_1125917_33.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/d3d28935-b3ad-45f2-8fa9-779473e974d7_isrForCrosstalkSources_1125917_33.stderr parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash d248d93eb4cda3ff7925cf07dda06567 parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/4b61d1b7-a5cc-4027-b0bd-7ce58a718354_isrForCrosstalkSources_1125903_3.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/4b61d1b7-a5cc-4027-b0bd-7ce58a718354_isrForCrosstalkSources_1125903_3.stderr parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash 3eac9ece03e59e95f2aabda73aa6fb29 parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/403f0e41-e520-4062-8af9-580ea7b9c180_isrForCrosstalkSources_1125918_53.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/403f0e41-e520-4062-8af9-580ea7b9c180_isrForCrosstalkSources_1125918_53.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash 9678ed4da6401806ef12f6d9ec139b96 parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/6d59bb72-d1bf-4b86-ab0b-93ac2e732856_isrForCrosstalkSources_1125898_14.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/6d59bb72-d1bf-4b86-ab0b-93ac2e732856_isrForCrosstalkSources_1125898_14.stderr parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash b205af48110cf9466b545d62cb565019 parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/0bec6ca9-3bf4-43d1-aebc-cda1e6488614_isrForCrosstalkSources_1125944_24.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/0bec6ca9-3bf4-43d1-aebc-cda1e6488614_isrForCrosstalkSources_1125944_24.stderr parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash 04442d79422d0d24aa0d251fea2ae2d4 parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/0242544f-785c-4882-9dde-1c15edfe8094_isrForCrosstalkSources_1125912_15.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/0242544f-785c-4882-9dde-1c15edfe8094_isrForCrosstalkSources_1125912_15.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 126625a47eec58d9d0425fdc91d0e556 parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/1ea81f94-db4c-4010-8d7b-3200e5fbe41e_isrForCrosstalkSources_1125937_31.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/1ea81f94-db4c-4010-8d7b-3200e5fbe41e_isrForCrosstalkSources_1125937_31.stderr parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash 5d8b3c745b4a6d33eedd9ec3d46dd53f parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a9501c5a-b263-4b44-bb71-12df24b98a59_isrForCrosstalkSources_1125937_61.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a9501c5a-b263-4b44-bb71-12df24b98a59_isrForCrosstalkSources_1125937_61.stderr parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash 8ae8bd126e2aa1337b2db22033e48440 parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/7bbe2948-7dac-4fbc-adca-5414405da6a8_isrForCrosstalkSources_1125906_28.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/7bbe2948-7dac-4fbc-adca-5414405da6a8_isrForCrosstalkSources_1125906_28.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash 57bfe241de1d9687cc55ce6e996fb375 parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/9828268b-4d7b-45e7-8dc9-d4f40a37754b_isrForCrosstalkSources_1125895_8.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/9828268b-4d7b-45e7-8dc9-d4f40a37754b_isrForCrosstalkSources_1125895_8.stderr parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash 3a150ee420936cbb95aa99cb0ae62fc0 parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/67ccf0fe-54e0-4eb6-958a-a28f7badaee0_isrForCrosstalkSources_1125942_30.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/67ccf0fe-54e0-4eb6-958a-a28f7badaee0_isrForCrosstalkSources_1125942_30.stderr parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash 41c03da7c8aa2e1cf7b604df0b7f600e parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/b1e4f637-d440-4115-ac99-100f1b993bd7_isrForCrosstalkSources_1125905_59.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/b1e4f637-d440-4115-ac99-100f1b993bd7_isrForCrosstalkSources_1125905_59.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash f631811fdb19d4f3d4e281eaa6ce743f parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/23cd4be0-c3b5-4bf9-9860-17225dd604dc_isrForCrosstalkSources_1125939_14.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/23cd4be0-c3b5-4bf9-9860-17225dd604dc_isrForCrosstalkSources_1125939_14.stderr parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash 28e3ca1fd4dcd571ee66601a5b868c0d parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/83298bbb-22f3-4938-af7e-46a687d77acc_isrForCrosstalkSources_1125925_60.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/83298bbb-22f3-4938-af7e-46a687d77acc_isrForCrosstalkSources_1125925_60.stderr parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash 35526b711b559415570f6d8bb7e382dd parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/fc35e5f6-d8c8-44dd-b394-2cfa1a837d88_isrForCrosstalkSources_1125910_55.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/fc35e5f6-d8c8-44dd-b394-2cfa1a837d88_isrForCrosstalkSources_1125910_55.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash e01dd06a2f3d52db9cc4c7b97818aa00 parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/8275782f-98d1-4929-9c2a-d53996efdb4f_isrForCrosstalkSources_1125946_39.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/8275782f-98d1-4929-9c2a-d53996efdb4f_isrForCrosstalkSources_1125946_39.stderr parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash 81563a08628975dc4d91bae374584fd0 parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/e91eb781-cbe0-45ca-a8de-3e103f40362e_isrForCrosstalkSources_1125936_26.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/e91eb781-cbe0-45ca-a8de-3e103f40362e_isrForCrosstalkSources_1125936_26.stderr parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash 86f83da29e14fe476b0d5968055ecf70 parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/26335c2e-bcf5-459a-8009-9c4ee28857e1_isrForCrosstalkSources_1125913_21.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/26335c2e-bcf5-459a-8009-9c4ee28857e1_isrForCrosstalkSources_1125913_21.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash e6650dc3c06c068f2caf22b57c30d259 parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/3205d968-6d13-47bc-993d-128bd46fa6ce_isrForCrosstalkSources_1125938_33.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/3205d968-6d13-47bc-993d-128bd46fa6ce_isrForCrosstalkSources_1125938_33.stderr parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 563 has memoization hash a01dd96f6c30d4d24305c3a5d1a0190d parsl.dataflow.memoization INFO: Task 563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:40 proc_lsst.multi:170 [INFO] [multi] job_id 16896386 proc_lsst.multi INFO: [multi] job_id 16896386 2024-03-07 15:06:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 15:06:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896386 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896386 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16896386 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-07 15:06:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-07 15:06:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 15:06:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/73b658b6-fdd2-4390-82ee-650ba81f681f_isrForCrosstalkSources_1125938_3.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/73b658b6-fdd2-4390-82ee-650ba81f681f_isrForCrosstalkSources_1125938_3.stderr parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash b3062b51b1d417dcc60180a9c58f96d8 parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/4116fa4f-ea41-4d14-82ed-cde4dba5bc10_isrForCrosstalkSources_1125902_12.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/4116fa4f-ea41-4d14-82ed-cde4dba5bc10_isrForCrosstalkSources_1125902_12.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash 56da911a43605c5b2b3e54a5559f427f parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/122a9042-cb03-4a7c-8b42-14fba991d0dd_isrForCrosstalkSources_1125918_17.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/122a9042-cb03-4a7c-8b42-14fba991d0dd_isrForCrosstalkSources_1125918_17.stderr parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash 6d37f88c9b8cf1ead1164a2425808884 parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/fa9eb24d-22a7-4018-87df-1cc3e16283c4_isrForCrosstalkSources_1125913_52.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/fa9eb24d-22a7-4018-87df-1cc3e16283c4_isrForCrosstalkSources_1125913_52.stderr parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash 03cd6a929fd4525a1c39d462969d3574 parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/70e361a6-3450-4925-aa43-bc9a388fbb7e_isrForCrosstalkSources_1125907_9.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/70e361a6-3450-4925-aa43-bc9a388fbb7e_isrForCrosstalkSources_1125907_9.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash a0389ca8866da525523c43e8af2454ab parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/04bfaa9e-7cf2-463a-95be-0257d1af52f2_isrForCrosstalkSources_1125934_2.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/04bfaa9e-7cf2-463a-95be-0257d1af52f2_isrForCrosstalkSources_1125934_2.stderr parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 569 has memoization hash 9062c7df05a01f80d8278d96302a56d3 parsl.dataflow.memoization INFO: Task 569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/db19c79d-d9a0-44ee-9b5e-a0e629bfbd20_isrForCrosstalkSources_1125894_19.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/db19c79d-d9a0-44ee-9b5e-a0e629bfbd20_isrForCrosstalkSources_1125894_19.stderr parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash e5665ec6b4186aa8bd1b1df3871ffea9 parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/56a92b19-f7ee-4cb4-a796-f81531d97f19_isrForCrosstalkSources_1125895_58.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/56a92b19-f7ee-4cb4-a796-f81531d97f19_isrForCrosstalkSources_1125895_58.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash 55870cf73f058997cc6fce9429bb1a03 parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/6529e91e-cee5-421c-bf43-8ce6946693d5_isrForCrosstalkSources_1125934_8.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/6529e91e-cee5-421c-bf43-8ce6946693d5_isrForCrosstalkSources_1125934_8.stderr parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash 68e7fb959c31c5de09b96eb375215ea9 parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/2f6afd46-03cf-4637-85e6-a188b5d5730e_isrForCrosstalkSources_1125939_37.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/2f6afd46-03cf-4637-85e6-a188b5d5730e_isrForCrosstalkSources_1125939_37.stderr parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash 0e52eec7b5c3ac8dca9a8efd3c6bd7ba parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/0641f819-aff0-46c9-b01b-8bac859cce8a_isrForCrosstalkSources_1125921_44.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/0641f819-aff0-46c9-b01b-8bac859cce8a_isrForCrosstalkSources_1125921_44.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash 1173a15134fc7ca09d4a6bedd8f93bc9 parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/de8a0887-45df-4dc0-a500-facf0fa5b69e_isrForCrosstalkSources_1125893_53.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/de8a0887-45df-4dc0-a500-facf0fa5b69e_isrForCrosstalkSources_1125893_53.stderr parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 575 has memoization hash d3e7d1ead79033b6a7147c1f90fc0d96 parsl.dataflow.memoization INFO: Task 575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e33af064-d698-41d4-b000-31da3aae9600_isrForCrosstalkSources_1125929_5.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e33af064-d698-41d4-b000-31da3aae9600_isrForCrosstalkSources_1125929_5.stderr parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash 93d5700941a06d195f4194c96908137f parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/b4f5d44a-cbca-4184-bbf2-fd8f7daf988b_isrForCrosstalkSources_1125907_25.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/b4f5d44a-cbca-4184-bbf2-fd8f7daf988b_isrForCrosstalkSources_1125907_25.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash 7980cfdf7bf0c8eb0b7117e587fdc357 parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/4c10aaa9-cec2-4e10-b06f-37c9a67a9447_isrForCrosstalkSources_1125924_40.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/4c10aaa9-cec2-4e10-b06f-37c9a67a9447_isrForCrosstalkSources_1125924_40.stderr parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 578 has memoization hash 2a0ad7095f23df1f36c664f2a3a5f9a0 parsl.dataflow.memoization INFO: Task 578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/489eb157-43ff-4e54-be3a-6f6b1244b22c_isrForCrosstalkSources_1125899_40.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/489eb157-43ff-4e54-be3a-6f6b1244b22c_isrForCrosstalkSources_1125899_40.stderr parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash 1cc05cbd09aacf8d3161c5b132fcff33 parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/8d528bb9-3294-4825-babf-76abdadb0394_isrForCrosstalkSources_1125907_2.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/8d528bb9-3294-4825-babf-76abdadb0394_isrForCrosstalkSources_1125907_2.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash 25e1fe53352205a2a6f31e9cd1769a42 parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/c6d73bad-d701-4698-a776-5e78715f6ff0_isrForCrosstalkSources_1125899_38.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/c6d73bad-d701-4698-a776-5e78715f6ff0_isrForCrosstalkSources_1125899_38.stderr parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash 0a99189230d852a83c6932ed14da461f parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a25cb8e0-9a1c-48a4-b4e2-1ed482f1bf98_isrForCrosstalkSources_1125921_7.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a25cb8e0-9a1c-48a4-b4e2-1ed482f1bf98_isrForCrosstalkSources_1125921_7.stderr parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash 8eb37052b683d61b567d2d3c7a755745 parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/114b86a5-12ec-4834-a55c-b926f848e340_isrForCrosstalkSources_1125899_6.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/114b86a5-12ec-4834-a55c-b926f848e340_isrForCrosstalkSources_1125899_6.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash 6bfa2e587a5bba958301ff2cb9cc8fbe parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/b4e174b6-bd11-4132-a62b-edd0dfb47b28_isrForCrosstalkSources_1125936_49.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/b4e174b6-bd11-4132-a62b-edd0dfb47b28_isrForCrosstalkSources_1125936_49.stderr parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 584 has memoization hash 7da0abc2b3e5ff6e1597d42c44d53cf1 parsl.dataflow.memoization INFO: Task 584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/bda2ceb9-b533-4547-80b4-449291e15a7f_isrForCrosstalkSources_1125916_58.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/bda2ceb9-b533-4547-80b4-449291e15a7f_isrForCrosstalkSources_1125916_58.stderr parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash 5630ce6479a2078f97f3d1cc28bc7f4b parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/ffffd5dc-92c1-41d9-91f2-1cffe42235de_isrForCrosstalkSources_1125892_35.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/ffffd5dc-92c1-41d9-91f2-1cffe42235de_isrForCrosstalkSources_1125892_35.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash 7d115167f04a999253fd03b796814609 parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/216a98d9-2aeb-43c4-9847-4f46159326c1_isrForCrosstalkSources_1125935_11.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/216a98d9-2aeb-43c4-9847-4f46159326c1_isrForCrosstalkSources_1125935_11.stderr parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash 84032aba461701aacbf360c7d74ec255 parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/0918d60e-0805-45de-8219-04451dd735f9_isrForCrosstalkSources_1125915_10.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/0918d60e-0805-45de-8219-04451dd735f9_isrForCrosstalkSources_1125915_10.stderr parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash 4e8dbe11ac4cb0191fdfe5031dbe94ef parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/756883fc-9048-48dd-b10a-0cb0fc7c7de7_isrForCrosstalkSources_1125892_37.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/756883fc-9048-48dd-b10a-0cb0fc7c7de7_isrForCrosstalkSources_1125892_37.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash a20e42bdf8d8e3207fa445b0a119ef16 parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/4e6ee0ab-41f7-476a-ac5d-21175ee48989_isrForCrosstalkSources_1125909_12.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/4e6ee0ab-41f7-476a-ac5d-21175ee48989_isrForCrosstalkSources_1125909_12.stderr parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 590 has memoization hash 2f26fa6dcd5d87c585aa956545287955 parsl.dataflow.memoization INFO: Task 590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/4077a22f-d85d-422f-bfac-495dabfab365_isrForCrosstalkSources_1125946_45.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/4077a22f-d85d-422f-bfac-495dabfab365_isrForCrosstalkSources_1125946_45.stderr parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash 789bbe7d272b00a5c518990e539bf542 parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ff361bb0-236a-479e-89ee-6b442ad1c7c1_isrForCrosstalkSources_1125918_33.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ff361bb0-236a-479e-89ee-6b442ad1c7c1_isrForCrosstalkSources_1125918_33.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash 2c9b0444d5972a4cd02a69b0aca143a9 parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/f14f624a-ff79-4b8a-acb4-de04f2008869_isrForCrosstalkSources_1125927_30.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/f14f624a-ff79-4b8a-acb4-de04f2008869_isrForCrosstalkSources_1125927_30.stderr parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 593 has memoization hash 977b3f177d6f83c767bf0a3d8924dc88 parsl.dataflow.memoization INFO: Task 593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/ad1e230d-784c-421f-9c2d-00ba3e89f3d3_isrForCrosstalkSources_1125893_61.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/ad1e230d-784c-421f-9c2d-00ba3e89f3d3_isrForCrosstalkSources_1125893_61.stderr parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash 309c695f6119cc7fa5b220b7798e3dd4 parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/c6f70afe-a821-4712-afdf-4a4aa8260c34_isrForCrosstalkSources_1125912_46.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/c6f70afe-a821-4712-afdf-4a4aa8260c34_isrForCrosstalkSources_1125912_46.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash 552a8ed995fe255f720c638fa709ff2d parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/4ba1c79a-02a1-4702-be3f-602ecd374cd5_isrForCrosstalkSources_1125939_17.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/4ba1c79a-02a1-4702-be3f-602ecd374cd5_isrForCrosstalkSources_1125939_17.stderr parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 596 has memoization hash de8a6b0ff03dd1f886dc67f31a831d10 parsl.dataflow.memoization INFO: Task 596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/278cf566-947b-4ae1-a728-4564a996cde7_isrForCrosstalkSources_1125929_24.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/278cf566-947b-4ae1-a728-4564a996cde7_isrForCrosstalkSources_1125929_24.stderr parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 597 has memoization hash ed425979e65923d444d915cf0b6c8ff5 parsl.dataflow.memoization INFO: Task 597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/de8799d8-20d3-4c9d-b247-3509bde0dfd8_isrForCrosstalkSources_1125909_32.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/de8799d8-20d3-4c9d-b247-3509bde0dfd8_isrForCrosstalkSources_1125909_32.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash 2701146bf0193ad2481e8804f5cf387a parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/c09e6a9d-0bac-475b-b673-96d16db269f4_isrForCrosstalkSources_1125915_22.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/c09e6a9d-0bac-475b-b673-96d16db269f4_isrForCrosstalkSources_1125915_22.stderr parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 599 has memoization hash 3c1cb7924f13b7748aed94a19ab4d79e parsl.dataflow.memoization INFO: Task 599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2665c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/ee658c11-0c8b-4f2c-b39f-4c55b780ab64_isrForCrosstalkSources_1125891_45.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/ee658c11-0c8b-4f2c-b39f-4c55b780ab64_isrForCrosstalkSources_1125891_45.stderr parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash 134406582bb4c2a172d6f098ccb1d1cc parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/b82d477a-20fc-40b7-9747-946a3dbfbf1b_isrForCrosstalkSources_1125911_3.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/b82d477a-20fc-40b7-9747-946a3dbfbf1b_isrForCrosstalkSources_1125911_3.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash a491971e5ea7de76fbc8ff2532dafeb7 parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/47bcc5fa-dbed-48fe-966f-21e1afd81d02_isrForCrosstalkSources_1125939_1.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/47bcc5fa-dbed-48fe-966f-21e1afd81d02_isrForCrosstalkSources_1125939_1.stderr parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash 9e6cd41aaefbf8ef2662024907df238d parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/74fd87d4-1e4c-446b-939e-4a77c118a192_isrForCrosstalkSources_1125919_15.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/74fd87d4-1e4c-446b-939e-4a77c118a192_isrForCrosstalkSources_1125919_15.stderr parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash 84b3259f5fa1416d1ef3c45adb8ff00a parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/b85ccc6c-cb02-49df-b0fb-51f03c89d5e4_isrForCrosstalkSources_1125888_21.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/b85ccc6c-cb02-49df-b0fb-51f03c89d5e4_isrForCrosstalkSources_1125888_21.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash d43ffbbcc50a001fdaf097ebc19227e5 parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/eaf18f41-8662-444f-8ac5-b4fe23fac25a_isrForCrosstalkSources_1125906_48.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/eaf18f41-8662-444f-8ac5-b4fe23fac25a_isrForCrosstalkSources_1125906_48.stderr parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash c5c8b7425c1fdbe886f7b98cd582f062 parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/352c8190-88f6-444c-ae21-fa25d0b1029f_isrForCrosstalkSources_1125892_52.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/352c8190-88f6-444c-ae21-fa25d0b1029f_isrForCrosstalkSources_1125892_52.stderr parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash 9d771b4383d8696b8363a111ba0000e7 parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/903bc0b1-ae99-42dc-a5fb-ae58c4eace0d_isrForCrosstalkSources_1125943_53.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/903bc0b1-ae99-42dc-a5fb-ae58c4eace0d_isrForCrosstalkSources_1125943_53.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash 2cb6f7718f5fb362292037035b5c8e1b parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/9eefe7f1-bcff-45d9-b24c-7dab775af627_isrForCrosstalkSources_1125931_40.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/9eefe7f1-bcff-45d9-b24c-7dab775af627_isrForCrosstalkSources_1125931_40.stderr parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash 4b4f4df511bfbfc132947f643b1b7894 parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/a8e08cc1-fae7-4360-95f3-9e0d543109f4_isrForCrosstalkSources_1125929_49.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/a8e08cc1-fae7-4360-95f3-9e0d543109f4_isrForCrosstalkSources_1125929_49.stderr parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash b5a59d4af0d0dd7bd35dc47e9d9e0f5b parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/cbf6137e-b302-4f5d-968c-067e4e774a04_isrForCrosstalkSources_1125899_26.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/cbf6137e-b302-4f5d-968c-067e4e774a04_isrForCrosstalkSources_1125899_26.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash 73bf4088ea03e47d82a479bc64190213 parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/7b46b0ba-d9d2-4fdb-b6a4-296c73c92627_isrForCrosstalkSources_1125914_38.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/7b46b0ba-d9d2-4fdb-b6a4-296c73c92627_isrForCrosstalkSources_1125914_38.stderr parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 611 has memoization hash 647057a4f31bd1a21daa7e80d2fe9e43 parsl.dataflow.memoization INFO: Task 611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/28e44312-74f3-4a1e-9118-2e01f2613427_isrForCrosstalkSources_1125913_18.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/28e44312-74f3-4a1e-9118-2e01f2613427_isrForCrosstalkSources_1125913_18.stderr parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash 623eb2aff9c4b52373411d5cb9417f72 parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/7e92762a-1011-4917-a988-c9497a94fc49_isrForCrosstalkSources_1125926_62.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/7e92762a-1011-4917-a988-c9497a94fc49_isrForCrosstalkSources_1125926_62.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash 18a7a8ae4323e1d5265a19fd709ef321 parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/632e2e2e-ab8b-4659-9b15-d5ea5c74f23e_isrForCrosstalkSources_1125934_27.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/632e2e2e-ab8b-4659-9b15-d5ea5c74f23e_isrForCrosstalkSources_1125934_27.stderr parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 614 has memoization hash 2ede0c0c7f7b7164f149cdc05eb92fb7 parsl.dataflow.memoization INFO: Task 614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/c2c890a5-5c62-4678-acc5-445c6cbc1132_isrForCrosstalkSources_1125932_12.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/c2c890a5-5c62-4678-acc5-445c6cbc1132_isrForCrosstalkSources_1125932_12.stderr parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash 5570ffbaa6220232362fa204bce9ac6d parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/b3fabe6c-5fcd-4bb8-92b5-7d8d3f95a732_isrForCrosstalkSources_1125907_8.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/b3fabe6c-5fcd-4bb8-92b5-7d8d3f95a732_isrForCrosstalkSources_1125907_8.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash 8e2c9638891f75ebad0ae5b9f96aac19 parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d4ecde88-4633-469b-9182-a6ea7857b013_isrForCrosstalkSources_1125905_55.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d4ecde88-4633-469b-9182-a6ea7857b013_isrForCrosstalkSources_1125905_55.stderr parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash 9ab03d67355b22925c6a2174e7a40fde parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f64ac83b-9730-454d-9239-92dca3bf3a81_isrForCrosstalkSources_1125910_54.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f64ac83b-9730-454d-9239-92dca3bf3a81_isrForCrosstalkSources_1125910_54.stderr parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash 24acf727b16fae5dca6234d836b787b8 parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/cd5c9a3f-d613-426a-bc99-9cbab60b31a3_isrForCrosstalkSources_1125933_18.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/cd5c9a3f-d613-426a-bc99-9cbab60b31a3_isrForCrosstalkSources_1125933_18.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash 037a6b251cc49d75d3e3d0c1f962179d parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/14088ea3-a470-4287-86a3-f02987ae70ed_isrForCrosstalkSources_1125942_55.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/14088ea3-a470-4287-86a3-f02987ae70ed_isrForCrosstalkSources_1125942_55.stderr parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 620 has memoization hash d218166444a99a26fafbff9d5dcbc00d parsl.dataflow.memoization INFO: Task 620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/2dd13774-696b-4227-bc62-1a8c9ec833e6_isrForCrosstalkSources_1125907_16.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/2dd13774-696b-4227-bc62-1a8c9ec833e6_isrForCrosstalkSources_1125907_16.stderr parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash 6838217aed4df9582fc6697d36fc08bf parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/9b887f82-9ca0-4ac8-a616-d6e476fd2ec9_isrForCrosstalkSources_1125929_46.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/9b887f82-9ca0-4ac8-a616-d6e476fd2ec9_isrForCrosstalkSources_1125929_46.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash 2402e8a1e7fcc29264f72678a76c568e parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/e1e7a69a-99d0-4b92-adf9-663b24a3c869_isrForCrosstalkSources_1125897_25.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/e1e7a69a-99d0-4b92-adf9-663b24a3c869_isrForCrosstalkSources_1125897_25.stderr parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash ca102f620197d9802c911491b09d9dbc parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2662a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/3bf1311d-6e7e-434f-82b2-87369be7eaab_isrForCrosstalkSources_1125918_9.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/3bf1311d-6e7e-434f-82b2-87369be7eaab_isrForCrosstalkSources_1125918_9.stderr parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash 03d458e8a4492c7f6666b591699eaa8c parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/fa53cbb1-44a4-45f7-aa0c-be060775b53c_isrForCrosstalkSources_1125918_5.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/fa53cbb1-44a4-45f7-aa0c-be060775b53c_isrForCrosstalkSources_1125918_5.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash eb259027cc047f750e8c5f116506ef06 parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2640e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/1f4c3c81-2c75-406c-9c40-a5be296601cd_isrForCrosstalkSources_1125947_16.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/1f4c3c81-2c75-406c-9c40-a5be296601cd_isrForCrosstalkSources_1125947_16.stderr parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 626 has memoization hash 8b015335a860c6d1e32e41a1d57a1e15 parsl.dataflow.memoization INFO: Task 626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/3434b134-28d6-479b-8afb-30754c27c851_isrForCrosstalkSources_1125945_42.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/3434b134-28d6-479b-8afb-30754c27c851_isrForCrosstalkSources_1125945_42.stderr parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash 777c8bbb832366fe2d307db0604547a1 parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/60f9094c-9e2a-4d08-be09-101e8fb83cde_isrForCrosstalkSources_1125917_6.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/60f9094c-9e2a-4d08-be09-101e8fb83cde_isrForCrosstalkSources_1125917_6.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash db10b7d00848978d32a2113c543dfca6 parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2642c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/e828da5c-843c-4eb9-b3a7-48182eb4cf37_isrForCrosstalkSources_1125893_14.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/e828da5c-843c-4eb9-b3a7-48182eb4cf37_isrForCrosstalkSources_1125893_14.stderr parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 629 has memoization hash b008543cc9d28a82d3838d207fd1111f parsl.dataflow.memoization INFO: Task 629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/b2ac82ed-b5d8-4769-8b2f-80ef588289ca_isrForCrosstalkSources_1125947_2.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/b2ac82ed-b5d8-4769-8b2f-80ef588289ca_isrForCrosstalkSources_1125947_2.stderr parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash 9470ed7c5099f469d1bc1524372800ea parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2644a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/11f4da60-2540-4f5d-b1ca-790624ff4a52_isrForCrosstalkSources_1125900_60.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/11f4da60-2540-4f5d-b1ca-790624ff4a52_isrForCrosstalkSources_1125900_60.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash 55af819dc3f80ace60d13f9c743c6574 parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/bbe5a108-dc3b-4bc2-bfa6-ec16a39ea6ff_isrForCrosstalkSources_1125923_1.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/bbe5a108-dc3b-4bc2-bfa6-ec16a39ea6ff_isrForCrosstalkSources_1125923_1.stderr parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 632 has memoization hash c77d091a1c37cd3c44cf37d92a2a134d parsl.dataflow.memoization INFO: Task 632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:41 proc_lsst.multi:170 [INFO] [multi] job_id 16896387 proc_lsst.multi INFO: [multi] job_id 16896387 2024-03-07 15:06:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 15:06:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896387 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896387 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16896387 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-07 15:06:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-07 15:06:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 15:06:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/bede607b-db3f-4c78-9e6b-a59a79f2a155_isrForCrosstalkSources_1125928_32.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/bede607b-db3f-4c78-9e6b-a59a79f2a155_isrForCrosstalkSources_1125928_32.stderr parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash e45bdc12c890cbb0902bffced1915e5d parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8b1f1eb1-ed39-4371-8699-bfec2a85699e_isrForCrosstalkSources_1125891_19.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8b1f1eb1-ed39-4371-8699-bfec2a85699e_isrForCrosstalkSources_1125891_19.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 5cd8a12a2539571f2e298ae828d7a26c parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/6e43801b-c305-45de-bb12-a776bee4f63e_isrForCrosstalkSources_1125928_28.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/6e43801b-c305-45de-bb12-a776bee4f63e_isrForCrosstalkSources_1125928_28.stderr parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 635 has memoization hash 5ffa062a790bf0e62855378ff6d17404 parsl.dataflow.memoization INFO: Task 635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/ef6d7c7e-959c-4934-b32d-83d2093f5b6e_isrForCrosstalkSources_1125903_4.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/ef6d7c7e-959c-4934-b32d-83d2093f5b6e_isrForCrosstalkSources_1125903_4.stderr parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash 71f2f5f2daf8ac65b4f6e67941e83b07 parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2645e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/9a98830c-e8c5-4c35-83eb-444eb1e1837e_isrForCrosstalkSources_1125928_45.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/9a98830c-e8c5-4c35-83eb-444eb1e1837e_isrForCrosstalkSources_1125928_45.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash a90129749dca003018cba0aa1a5a6d6e parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/694a26bb-3478-4afd-a6ce-752bb4b24d4c_isrForCrosstalkSources_1125900_50.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/694a26bb-3478-4afd-a6ce-752bb4b24d4c_isrForCrosstalkSources_1125900_50.stderr parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash 54fb7f0a3cfcb5595260bfe211025198 parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/6a04c0e9-fbec-4357-998a-463db64eaa7d_isrForCrosstalkSources_1125917_43.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/6a04c0e9-fbec-4357-998a-463db64eaa7d_isrForCrosstalkSources_1125917_43.stderr parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash d439a009a2d1ba4ede2137d271faed72 parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/c165cd5c-f6d5-4838-8dc0-17c610198360_isrForCrosstalkSources_1125935_34.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/c165cd5c-f6d5-4838-8dc0-17c610198360_isrForCrosstalkSources_1125935_34.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash eca4db9e78662c10e536e1802354c9e6 parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/5224934c-04a6-49ec-8914-28bdb90475dc_isrForCrosstalkSources_1125902_32.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/5224934c-04a6-49ec-8914-28bdb90475dc_isrForCrosstalkSources_1125902_32.stderr parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 641 has memoization hash 9fe3416039cedb4a1d176168791ca88c parsl.dataflow.memoization INFO: Task 641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7ec64d66-ce0b-4ea9-9936-519d1ab2afe7_isrForCrosstalkSources_1125929_29.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7ec64d66-ce0b-4ea9-9936-519d1ab2afe7_isrForCrosstalkSources_1125929_29.stderr parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash ad32a785ed084cee19526bfe24a9e317 parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/5e20b04a-2c4c-4df3-be2b-66c38cb8759f_isrForCrosstalkSources_1125921_22.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/5e20b04a-2c4c-4df3-be2b-66c38cb8759f_isrForCrosstalkSources_1125921_22.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 643 has memoization hash a762c9fa46d3f53b2c88c23ee246afa1 parsl.dataflow.memoization INFO: Task 643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/c0e25158-f739-4be2-8ac7-bece00cd2617_isrForCrosstalkSources_1125942_52.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/c0e25158-f739-4be2-8ac7-bece00cd2617_isrForCrosstalkSources_1125942_52.stderr parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash d916a5b01857bd8277f573050f131998 parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2649a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/dbc9d125-d061-4418-908b-297abe954460_isrForCrosstalkSources_1125935_5.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/dbc9d125-d061-4418-908b-297abe954460_isrForCrosstalkSources_1125935_5.stderr parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash f065a00e18b59bbb015cf84fa1a5484d parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/33897794-2ef6-4888-afe1-913e4776c10b_isrForCrosstalkSources_1125899_28.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/33897794-2ef6-4888-afe1-913e4776c10b_isrForCrosstalkSources_1125899_28.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash 22cea8a089819bd6574a71f9b5b4548c parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/75f7d147-6341-48e0-b3c9-c4acded97e3b_isrForCrosstalkSources_1125913_27.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/75f7d147-6341-48e0-b3c9-c4acded97e3b_isrForCrosstalkSources_1125913_27.stderr parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash 1a7b7eea5713f685767f3f1b996cb72f parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/a0b04d21-fd02-4960-999d-0635cdf23a2f_isrForCrosstalkSources_1125946_48.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/a0b04d21-fd02-4960-999d-0635cdf23a2f_isrForCrosstalkSources_1125946_48.stderr parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash 8d1e15e9e8249582d1afcf438cbfcc4d parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/b01d9c90-fc8f-4f52-8f28-7df20ac76aa2_isrForCrosstalkSources_1125903_30.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/b01d9c90-fc8f-4f52-8f28-7df20ac76aa2_isrForCrosstalkSources_1125903_30.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash d5178efa5e99013b10303d435de5b89b parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/da5a024f-92a0-4d53-984a-00743d03fa90_isrForCrosstalkSources_1125924_12.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/da5a024f-92a0-4d53-984a-00743d03fa90_isrForCrosstalkSources_1125924_12.stderr parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash 00410611cf980c99745bd8dc1cc62448 parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/a62fe34f-5663-4b43-883e-c4e1d4dceb1d_isrForCrosstalkSources_1125931_8.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/a62fe34f-5663-4b43-883e-c4e1d4dceb1d_isrForCrosstalkSources_1125931_8.stderr parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash 8094c6500fbdbff5c126e78cd4f5a6e7 parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a6acc7d5-e51e-4973-8de4-1cdb48ee99a0_isrForCrosstalkSources_1125900_31.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a6acc7d5-e51e-4973-8de4-1cdb48ee99a0_isrForCrosstalkSources_1125900_31.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash fd37ca56cc7586c2edf4b96a0293d00b parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/c459f0f8-7609-4871-85cf-138fc827c1a9_isrForCrosstalkSources_1125897_24.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/c459f0f8-7609-4871-85cf-138fc827c1a9_isrForCrosstalkSources_1125897_24.stderr parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash 86d484104eb67e077a2b26f9546307a7 parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d264e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/4cbf1b39-b1c5-454c-b097-9b87f6c39af0_isrForCrosstalkSources_1125944_48.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/4cbf1b39-b1c5-454c-b097-9b87f6c39af0_isrForCrosstalkSources_1125944_48.stderr parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash 3ea2a5ceead89834cf73470a36686307 parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/f2e24ea0-5899-4c78-abb7-aa12d49f2167_isrForCrosstalkSources_1125938_44.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/f2e24ea0-5899-4c78-abb7-aa12d49f2167_isrForCrosstalkSources_1125938_44.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash 32197bf767b097afb84a92f3ed08343f parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/90ab9046-5499-4317-b54e-f6c446840c88_isrForCrosstalkSources_1125910_44.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/90ab9046-5499-4317-b54e-f6c446840c88_isrForCrosstalkSources_1125910_44.stderr parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 656 has memoization hash 3865e0190d76f026d86a9751e4194254 parsl.dataflow.memoization INFO: Task 656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/e145070b-e4af-47b2-a6ff-2e268d20279d_isrForCrosstalkSources_1125918_29.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/e145070b-e4af-47b2-a6ff-2e268d20279d_isrForCrosstalkSources_1125918_29.stderr parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash 3f876ddda257134a9a5b358535fab1f5 parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/8cb1ef68-1d87-4500-b2d1-d36e2dda1d3e_isrForCrosstalkSources_1125901_58.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/8cb1ef68-1d87-4500-b2d1-d36e2dda1d3e_isrForCrosstalkSources_1125901_58.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash be8c4d47b79e1b7b9705284a013a69b8 parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2663e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/44d1436d-6962-4652-93e0-137245cd32c8_isrForCrosstalkSources_1125902_19.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/44d1436d-6962-4652-93e0-137245cd32c8_isrForCrosstalkSources_1125902_19.stderr parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash 017063ad028807d74f02a21db93f57cb parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2647c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/1348edcd-ae70-4c4a-8be5-be85895a8316_isrForCrosstalkSources_1125893_8.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/1348edcd-ae70-4c4a-8be5-be85895a8316_isrForCrosstalkSources_1125893_8.stderr parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash 72cb71a8fef6e82a3166c2e935966995 parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/01252630-c733-4eb9-a4e6-a1a910098fa3_isrForCrosstalkSources_1125932_52.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/01252630-c733-4eb9-a4e6-a1a910098fa3_isrForCrosstalkSources_1125932_52.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash 59c13ddb3eaa9b5d50728f54f140d096 parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2660c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/d3aad5ad-29e2-4182-beb4-3f89bc4a9927_isrForCrosstalkSources_1125926_25.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/d3aad5ad-29e2-4182-beb4-3f89bc4a9927_isrForCrosstalkSources_1125926_25.stderr parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash af35641ea225eed7797ca2daff4a1826 parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/900d159c-e297-4c0c-adbb-24ba9c1a84e9_isrForCrosstalkSources_1125893_7.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/900d159c-e297-4c0c-adbb-24ba9c1a84e9_isrForCrosstalkSources_1125893_7.stderr parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash a5aa07423499a6ef22a378371a31cbee parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/723b9f26-2b4e-418b-808f-4d3f8b4398ff_isrForCrosstalkSources_1125917_1.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/723b9f26-2b4e-418b-808f-4d3f8b4398ff_isrForCrosstalkSources_1125917_1.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash 3b5d878096be3128eec9f985969b6ffc parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/8b981db2-4805-40f6-8e13-3cbcd31ffda1_isrForCrosstalkSources_1125947_17.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/8b981db2-4805-40f6-8e13-3cbcd31ffda1_isrForCrosstalkSources_1125947_17.stderr parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 665 has memoization hash 939eb5c83864383f13263eed1d2f926a parsl.dataflow.memoization INFO: Task 665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/aefb86fe-7e81-4a87-9036-8f8dc5a07114_isrForCrosstalkSources_1125922_35.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/aefb86fe-7e81-4a87-9036-8f8dc5a07114_isrForCrosstalkSources_1125922_35.stderr parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 666 has memoization hash 96ba927ecaca24b724b59849c6901e3e parsl.dataflow.memoization INFO: Task 666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/907bdbba-245b-460a-8cfc-f5384f3dd506_isrForCrosstalkSources_1125922_17.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/907bdbba-245b-460a-8cfc-f5384f3dd506_isrForCrosstalkSources_1125922_17.stderr parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash ae5f27c705e1cb7a2aa05ab6a2ba6b0a parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/e272eea1-188a-492e-a49e-f0a6b7bafbdf_isrForCrosstalkSources_1125938_37.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/e272eea1-188a-492e-a49e-f0a6b7bafbdf_isrForCrosstalkSources_1125938_37.stderr parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash 0573d41e0af18d21453587fc98410e67 parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/48945244-3b3a-4f7b-b389-3b523c8bc99f_isrForCrosstalkSources_1125937_36.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/48945244-3b3a-4f7b-b389-3b523c8bc99f_isrForCrosstalkSources_1125937_36.stderr parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash c936802fdb96717fb0c441d4d5c78332 parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/895927f3-3524-4693-a0f0-3bc9fb1b90d1_isrForCrosstalkSources_1125906_30.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/895927f3-3524-4693-a0f0-3bc9fb1b90d1_isrForCrosstalkSources_1125906_30.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash 51536b4aa3a43c5a181880aa9f73e243 parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/db13a9a6-420f-4a62-9de0-1a9011b89246_isrForCrosstalkSources_1125919_41.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/db13a9a6-420f-4a62-9de0-1a9011b89246_isrForCrosstalkSources_1125919_41.stderr parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 45f49f171fe777523a1a5d37b0156777 parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/3cd5944c-c2ad-4f00-b3b9-53158e922d83_isrForCrosstalkSources_1125939_45.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/3cd5944c-c2ad-4f00-b3b9-53158e922d83_isrForCrosstalkSources_1125939_45.stderr parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash 477bebd7b47e7f5e065c993d9e744d6a parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/943df236-e5c3-45d2-9d6c-b381729c76df_isrForCrosstalkSources_1125941_5.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/943df236-e5c3-45d2-9d6c-b381729c76df_isrForCrosstalkSources_1125941_5.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash a85c9a043d6d9b171569552255ab4432 parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f4b54922-192b-4ffd-8832-7363a7d7e570_isrForCrosstalkSources_1125922_29.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f4b54922-192b-4ffd-8832-7363a7d7e570_isrForCrosstalkSources_1125922_29.stderr parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 674 has memoization hash 8bcbfb47c9fa824fd4a2554d1404b754 parsl.dataflow.memoization INFO: Task 674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/e1ca9711-c08e-4ab9-b27b-edddd36e8f91_isrForCrosstalkSources_1125914_11.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/e1ca9711-c08e-4ab9-b27b-edddd36e8f91_isrForCrosstalkSources_1125914_11.stderr parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash cc200b4d219c4502438b0ef6bcdf6738 parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/c6d9044a-6b1b-4451-8c52-2ca0e54e22e6_isrForCrosstalkSources_1125925_3.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/c6d9044a-6b1b-4451-8c52-2ca0e54e22e6_isrForCrosstalkSources_1125925_3.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash 682fd575ef5ba9287590931cc2106340 parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/24174e74-97aa-4fa7-9ff4-f545d7a5e863_isrForCrosstalkSources_1125922_32.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/24174e74-97aa-4fa7-9ff4-f545d7a5e863_isrForCrosstalkSources_1125922_32.stderr parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash 1a2d5533a49c911e4f2687bf67eace9c parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2658a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/4db11ff5-21c4-44c7-8797-fd5fe49e3f5a_isrForCrosstalkSources_1125934_34.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/4db11ff5-21c4-44c7-8797-fd5fe49e3f5a_isrForCrosstalkSources_1125934_34.stderr parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash 8fe394745626d0d9b820086b822e92c0 parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2659e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/7956e711-9ec7-4e9e-abee-e869780ed09a_isrForCrosstalkSources_1125892_13.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/7956e711-9ec7-4e9e-abee-e869780ed09a_isrForCrosstalkSources_1125892_13.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash d241dabd5251eee05a4cbddb7f20cca9 parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/dff8a3df-4fdc-4500-97b3-49d3ea823d72_isrForCrosstalkSources_1125946_33.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/dff8a3df-4fdc-4500-97b3-49d3ea823d72_isrForCrosstalkSources_1125946_33.stderr parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash 452d67cbc1db36352de9363abb73081b parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/aebb9ff5-5efa-4fed-a117-240daa5a061a_isrForCrosstalkSources_1125906_50.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/aebb9ff5-5efa-4fed-a117-240daa5a061a_isrForCrosstalkSources_1125906_50.stderr parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash de4c670a335b04d736b741a3ae549f08 parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/227673b5-68c0-43e1-b245-3ec5a54fc69d_isrForCrosstalkSources_1125919_28.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/227673b5-68c0-43e1-b245-3ec5a54fc69d_isrForCrosstalkSources_1125919_28.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash 3252e0e17a065dccd7c21fcdfa7e311e parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/13f53b0a-52a2-42fa-875f-7e02873e4ddb_isrForCrosstalkSources_1125913_12.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/13f53b0a-52a2-42fa-875f-7e02873e4ddb_isrForCrosstalkSources_1125913_12.stderr parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 683 has memoization hash f00993645aa3a322d6e118809bf0e4fd parsl.dataflow.memoization INFO: Task 683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2656c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/50dcdd19-d00a-4cd1-a373-e4724cde3f12_isrForCrosstalkSources_1125905_48.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/50dcdd19-d00a-4cd1-a373-e4724cde3f12_isrForCrosstalkSources_1125905_48.stderr parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash aaa82791ca58242ce00ace1f29ab4f66 parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2654e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d220fdbf-a227-44f4-a256-9cf326e7dddc_isrForCrosstalkSources_1125940_11.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d220fdbf-a227-44f4-a256-9cf326e7dddc_isrForCrosstalkSources_1125940_11.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash 7e5732c0771a55540225c7b8e5359cc7 parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/6c251bf5-890d-4ba1-a4f8-f53f1073688f_isrForCrosstalkSources_1125921_17.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/6c251bf5-890d-4ba1-a4f8-f53f1073688f_isrForCrosstalkSources_1125921_17.stderr parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 686 has memoization hash 00fadabb1ef07c572577d172a626fcbe parsl.dataflow.memoization INFO: Task 686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2653a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/5ff75467-3c29-4b7b-acbb-2f6d4c6769c3_isrForCrosstalkSources_1125946_5.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/5ff75467-3c29-4b7b-acbb-2f6d4c6769c3_isrForCrosstalkSources_1125946_5.stderr parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash 040a0bbaef90ebdaa412299003d9f094 parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/f20cff46-4925-4f30-a696-24e9238eabf8_isrForCrosstalkSources_1125935_28.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/f20cff46-4925-4f30-a696-24e9238eabf8_isrForCrosstalkSources_1125935_28.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash d449055026f8a92f1a1b8a10b5698b59 parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/d20c8fb1-4fd9-4e18-bc40-78819570b4c1_isrForCrosstalkSources_1125889_26.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/d20c8fb1-4fd9-4e18-bc40-78819570b4c1_isrForCrosstalkSources_1125889_26.stderr parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 689 has memoization hash 11670301a704586499837a71832068bc parsl.dataflow.memoization INFO: Task 689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/991914a0-f974-4f14-8cce-767aa4da6cf2_isrForCrosstalkSources_1125932_58.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/991914a0-f974-4f14-8cce-767aa4da6cf2_isrForCrosstalkSources_1125932_58.stderr parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash 9a2c4af316322e944f496fa769a40092 parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2667a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/fe921c47-c2cb-47fd-a254-bc5575094594_isrForCrosstalkSources_1125930_16.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/fe921c47-c2cb-47fd-a254-bc5575094594_isrForCrosstalkSources_1125930_16.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash 8673ddc87896fed23fc3253b53420540 parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/4d8bf06b-f796-4c95-9608-af680c4ca103_isrForCrosstalkSources_1125901_61.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/4d8bf06b-f796-4c95-9608-af680c4ca103_isrForCrosstalkSources_1125901_61.stderr parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 692 has memoization hash 6c3e595b5a6b5955df3e285b85c53c40 parsl.dataflow.memoization INFO: Task 692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/95071f16-2744-42fd-8f5e-db3a18d04db4_isrForCrosstalkSources_1125903_38.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/95071f16-2744-42fd-8f5e-db3a18d04db4_isrForCrosstalkSources_1125903_38.stderr parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash 4e6e5257b5a873f6eab341148c2257f7 parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2651c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/f4691488-7fa0-4bb8-9c67-f1cfa66ba7c6_isrForCrosstalkSources_1125899_4.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/f4691488-7fa0-4bb8-9c67-f1cfa66ba7c6_isrForCrosstalkSources_1125899_4.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash 655a4185d41bd3a79b799d8774a7cff5 parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2668e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/4fc6af07-92de-4f2c-8f6f-ffb751f3cc4f_isrForCrosstalkSources_1125942_22.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/4fc6af07-92de-4f2c-8f6f-ffb751f3cc4f_isrForCrosstalkSources_1125942_22.stderr parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 695 has memoization hash 8a6def2eafc86e8c1b37c2ab53e53e1c parsl.dataflow.memoization INFO: Task 695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/1539bc16-1755-477a-a312-82a17488c9ef_isrForCrosstalkSources_1125942_62.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/1539bc16-1755-477a-a312-82a17488c9ef_isrForCrosstalkSources_1125942_62.stderr parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash 3caf66a86c6cab2f56440d2bf8aeeb57 parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a6845c39-e197-4ddf-b224-0e91f647f6a3_isrForCrosstalkSources_1125933_43.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a6845c39-e197-4ddf-b224-0e91f647f6a3_isrForCrosstalkSources_1125933_43.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash ced73ce03545586b9b4b61d7581a4a77 parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/dbb6769e-cc60-4696-94f2-3644c7a810e1_isrForCrosstalkSources_1125905_33.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/dbb6769e-cc60-4696-94f2-3644c7a810e1_isrForCrosstalkSources_1125905_33.stderr parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 698 has memoization hash 794b68e27cccb50251f38ed2e63996e2 parsl.dataflow.memoization INFO: Task 698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/d3a172fc-9c41-4ede-bcbc-c44bc8458ea7_isrForCrosstalkSources_1125945_30.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/d3a172fc-9c41-4ede-bcbc-c44bc8458ea7_isrForCrosstalkSources_1125945_30.stderr parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash 0ec62e122d554c4a48e19ed9da148217 parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/42d543c7-2c52-4e1a-9762-7923c45274ed_isrForCrosstalkSources_1125945_53.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/42d543c7-2c52-4e1a-9762-7923c45274ed_isrForCrosstalkSources_1125945_53.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash 933ac74a0796bc0fd2e4de0e75a04743 parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/a829ba6f-ef82-4805-808d-eed6c28a257e_isrForCrosstalkSources_1125927_13.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/a829ba6f-ef82-4805-808d-eed6c28a257e_isrForCrosstalkSources_1125927_13.stderr parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash 133d02467ed589a1d43c45637e7e0b8f parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/3caef4ad-6e3d-4965-9e88-edf215a19a98_isrForCrosstalkSources_1125891_14.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/3caef4ad-6e3d-4965-9e88-edf215a19a98_isrForCrosstalkSources_1125891_14.stderr parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash 70251b62c50e9aeddc9e341aeb9a6d7e parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/a611dd9e-23ba-4b1a-96cf-3031df86acd6_isrForCrosstalkSources_1125888_29.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/a611dd9e-23ba-4b1a-96cf-3031df86acd6_isrForCrosstalkSources_1125888_29.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash 885c2b7b2c409fe3ed17dd92a58b5d0d parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/4b3d29cf-db80-4e90-a972-ebc14870513d_isrForCrosstalkSources_1125891_12.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/4b3d29cf-db80-4e90-a972-ebc14870513d_isrForCrosstalkSources_1125891_12.stderr parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash bce28671393b1c12c77c14fbf30b9d33 parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:42 proc_lsst.multi:170 [INFO] [multi] job_id 16896388 proc_lsst.multi INFO: [multi] job_id 16896388 2024-03-07 15:06:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 15:06:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896388 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896388 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16896388 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-07 15:06:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-07 15:06:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 15:06:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/c67d914d-8a2d-488b-ac07-4e4cf9152146_isrForCrosstalkSources_1125945_21.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/c67d914d-8a2d-488b-ac07-4e4cf9152146_isrForCrosstalkSources_1125945_21.stderr parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash 4c567873298f67a2a82c76387b45a31f parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/d00a3b80-6683-466b-865d-d8732fb6f245_isrForCrosstalkSources_1125897_6.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/d00a3b80-6683-466b-865d-d8732fb6f245_isrForCrosstalkSources_1125897_6.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash 590acdc768ce4ab48d62b8f7ebeea696 parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2f756ded-94d8-404f-9307-6c565df5e351_isrForCrosstalkSources_1125895_6.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2f756ded-94d8-404f-9307-6c565df5e351_isrForCrosstalkSources_1125895_6.stderr parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash e9326af715963c042b51b2ef84751a87 parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/dc8432fc-2513-4322-acb7-c42a22b08c8e_isrForCrosstalkSources_1125933_8.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/dc8432fc-2513-4322-acb7-c42a22b08c8e_isrForCrosstalkSources_1125933_8.stderr parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash 895e4c1401cb68d7ba2533c44680577f parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d265e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 709 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/746b81b9-9e59-46da-949b-4b5e5e920ee7_isrForCrosstalkSources_1125916_12.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/746b81b9-9e59-46da-949b-4b5e5e920ee7_isrForCrosstalkSources_1125916_12.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash f3246698bb058862250445766d76c132 parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/f6b21adb-64ae-4c80-b45f-9800069fe19a_isrForCrosstalkSources_1125918_24.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/f6b21adb-64ae-4c80-b45f-9800069fe19a_isrForCrosstalkSources_1125918_24.stderr parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash 56d059cd348e7e7c5f5bbdf0c5d784ca parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/00732f5e-95ba-466b-afd8-f81b4a7cd27a_isrForCrosstalkSources_1125891_34.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/00732f5e-95ba-466b-afd8-f81b4a7cd27a_isrForCrosstalkSources_1125891_34.stderr parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash 16cb7020933bbfcb92f48975f8635a98 parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2672e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/106ba483-46f6-402b-8144-649a2fe6047f_isrForCrosstalkSources_1125932_42.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/106ba483-46f6-402b-8144-649a2fe6047f_isrForCrosstalkSources_1125932_42.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash f8929b47efd7bd37cc1582993b671ba4 parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/84fa837c-52f9-4216-8ce5-224a83613507_isrForCrosstalkSources_1125930_27.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/84fa837c-52f9-4216-8ce5-224a83613507_isrForCrosstalkSources_1125930_27.stderr parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 713 has memoization hash 470c1e2b9e468e7cd91405b636b7e447 parsl.dataflow.memoization INFO: Task 713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/11c55d90-ec2c-4058-a7e7-3617a610e318_isrForCrosstalkSources_1125933_40.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/11c55d90-ec2c-4058-a7e7-3617a610e318_isrForCrosstalkSources_1125933_40.stderr parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash 7d9741e0a18a7b6d23c1c2ffcb44750d parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/345576a6-1450-4a7c-b315-3e930c0bb1f4_isrForCrosstalkSources_1125934_33.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/345576a6-1450-4a7c-b315-3e930c0bb1f4_isrForCrosstalkSources_1125934_33.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 998b0c337001affa4cad67702c938246 parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/1983f850-10d5-402a-ae63-194dc2fc048e_isrForCrosstalkSources_1125931_33.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/1983f850-10d5-402a-ae63-194dc2fc048e_isrForCrosstalkSources_1125931_33.stderr parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 8b47544a9aceb3c5e3870cefdf9c1aa7 parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/2f4a1cec-6c8b-49e1-8b92-b1f845ebd6d5_isrForCrosstalkSources_1125947_26.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/2f4a1cec-6c8b-49e1-8b92-b1f845ebd6d5_isrForCrosstalkSources_1125947_26.stderr parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash 912a7ceeab380beb2e6f5f25fb4f6f91 parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2674c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/420bde2e-bc97-4557-bd52-38ab3ac01b21_isrForCrosstalkSources_1125915_44.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/420bde2e-bc97-4557-bd52-38ab3ac01b21_isrForCrosstalkSources_1125915_44.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash d833303a387f5cc2f93a81f4d1cafbcb parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2676a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/dda2c83d-ed7d-4b26-9281-512677e84c84_isrForCrosstalkSources_1125939_49.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/dda2c83d-ed7d-4b26-9281-512677e84c84_isrForCrosstalkSources_1125939_49.stderr parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash b5cbdc3dac4a050fc5cb0a9c6f9c77d1 parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d0e7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/ff8ed4d2-9b77-48d5-973d-30638a5e9f04_isrForCrosstalkSources_1125917_44.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/ff8ed4d2-9b77-48d5-973d-30638a5e9f04_isrForCrosstalkSources_1125917_44.stderr parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash 7af4c80b4df61512caf6aba3329e5842 parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/fe8b18be-2a7a-42c1-bf82-e5748a4b2fa3_isrForCrosstalkSources_1125901_7.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/fe8b18be-2a7a-42c1-bf82-e5748a4b2fa3_isrForCrosstalkSources_1125901_7.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash 29afc7ca1a7ea7028748a35ba2c49110 parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/b3c045c2-00f6-4f12-b188-c124bf3eed30_isrForCrosstalkSources_1125908_16.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/b3c045c2-00f6-4f12-b188-c124bf3eed30_isrForCrosstalkSources_1125908_16.stderr parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash 933f120b6ddce53474f79ca34c6aa7a0 parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2679c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/83550407-ee95-472b-9dce-0e83a1c72c77_isrForCrosstalkSources_1125904_9.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/83550407-ee95-472b-9dce-0e83a1c72c77_isrForCrosstalkSources_1125904_9.stderr parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash 672dee9745f06e49498841ec3b2b71a7 parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/fabc9f51-f697-4cd1-b62f-e1b096491f24_isrForCrosstalkSources_1125900_20.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/fabc9f51-f697-4cd1-b62f-e1b096491f24_isrForCrosstalkSources_1125900_20.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash 8a293af78f12d05f1444ae6cf650f991 parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/97ade5a3-2ed0-4d4b-af06-ebc2bde4db6f_isrForCrosstalkSources_1125900_62.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/97ade5a3-2ed0-4d4b-af06-ebc2bde4db6f_isrForCrosstalkSources_1125900_62.stderr parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash a8a3353b1e6c7ab180e8e7cc5a624f01 parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/99c2d408-48ec-4366-9524-8312e0c56a9e_isrForCrosstalkSources_1125906_52.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/99c2d408-48ec-4366-9524-8312e0c56a9e_isrForCrosstalkSources_1125906_52.stderr parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash a79c9ba522c71f47298f531afa231a53 parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/dac2beb3-cf94-4bbd-86bf-4a93f1a827d8_isrForCrosstalkSources_1125929_20.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/dac2beb3-cf94-4bbd-86bf-4a93f1a827d8_isrForCrosstalkSources_1125929_20.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash 6546daf2cf71c129004425c7add1640f parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/5b0bcb2e-def8-42e5-9368-d9e5a07a8eb9_isrForCrosstalkSources_1125936_9.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/5b0bcb2e-def8-42e5-9368-d9e5a07a8eb9_isrForCrosstalkSources_1125936_9.stderr parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash fcee2582da20a55fd1b629106dea3dd7 parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/a06249a3-efc8-4b2d-81e2-856115be8c6c_isrForCrosstalkSources_1125902_8.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/a06249a3-efc8-4b2d-81e2-856115be8c6c_isrForCrosstalkSources_1125902_8.stderr parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash b4b73111f7222eeb8de98fd9a911aaa1 parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/238c8414-cd5d-48eb-aa93-d0e030917ea1_isrForCrosstalkSources_1125941_52.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/238c8414-cd5d-48eb-aa93-d0e030917ea1_isrForCrosstalkSources_1125941_52.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash dbde892750711bf53d5c054d8f78f875 parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/1cce7af8-8011-498f-abc3-f8e97ab57984_isrForCrosstalkSources_1125897_44.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/1cce7af8-8011-498f-abc3-f8e97ab57984_isrForCrosstalkSources_1125897_44.stderr parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash 511fcb09af38cc2277507b78fdaadbb8 parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/22a00018-7afd-4e9d-ace6-1ba98560b115_isrForCrosstalkSources_1125945_27.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/22a00018-7afd-4e9d-ace6-1ba98560b115_isrForCrosstalkSources_1125945_27.stderr parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash 8d4473c50cbec2cb36fac0798b99e6e7 parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/d9007c50-b50b-4541-8c2c-044285a99bce_isrForCrosstalkSources_1125914_43.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/d9007c50-b50b-4541-8c2c-044285a99bce_isrForCrosstalkSources_1125914_43.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash 3a2037b4b37524a69d541e8bfbd798d1 parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/7c953b72-b776-420e-8d35-e6ddd92f1584_isrForCrosstalkSources_1125946_25.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/7c953b72-b776-420e-8d35-e6ddd92f1584_isrForCrosstalkSources_1125946_25.stderr parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 734 has memoization hash b9cd9a3b12a5b2566695f3f8e510868f parsl.dataflow.memoization INFO: Task 734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:42 proc_lsst.multi:170 [INFO] [multi] job_id 16896389 proc_lsst.multi INFO: [multi] job_id 16896389 2024-03-07 15:06:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:06:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896389 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896389 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16896389 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-07 15:06:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-07 15:06:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:06:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/3cb2e033-a0d4-4b24-b893-c0bb7e664ce6_isrForCrosstalkSources_1125927_26.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/3cb2e033-a0d4-4b24-b893-c0bb7e664ce6_isrForCrosstalkSources_1125927_26.stderr parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 735 has memoization hash dd46b280449f0f4aa8ff536ef205ded2 parsl.dataflow.memoization INFO: Task 735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2671a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/4defcb48-6dee-4b68-b7b9-f3656aa46ac2_isrForCrosstalkSources_1125922_47.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/4defcb48-6dee-4b68-b7b9-f3656aa46ac2_isrForCrosstalkSources_1125922_47.stderr parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash d36a9da0d7c1aa1877269d77a8c7f3bb parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/1f26c640-e843-47c9-8160-12a592b9a7cc_isrForCrosstalkSources_1125913_56.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/1f26c640-e843-47c9-8160-12a592b9a7cc_isrForCrosstalkSources_1125913_56.stderr parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash 5031da1e08d16284e270d32dc31512d9 parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d2677e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/7febfae3-e83c-41c4-b992-f3ebcd5517da_isrForCrosstalkSources_1125905_58.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/7febfae3-e83c-41c4-b992-f3ebcd5517da_isrForCrosstalkSources_1125905_58.stderr parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash cebbfbb9d428b33f56122f85a1013d11 parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/6295614f-9241-4bbb-a8d9-8540426e082a_isrForCrosstalkSources_1125899_57.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/6295614f-9241-4bbb-a8d9-8540426e082a_isrForCrosstalkSources_1125899_57.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash 8e080894ecab500bcb5dc7a60bb5e7a2 parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/5d2ef4f0-a0f1-4fc9-9d05-c84efdeca7bb_isrForCrosstalkSources_1125938_4.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/5d2ef4f0-a0f1-4fc9-9d05-c84efdeca7bb_isrForCrosstalkSources_1125938_4.stderr parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash ef61a30ca3b00d4fd84b8b603ced0538 parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/00db2054-d53b-4313-9f38-4bffe912ed86_isrForCrosstalkSources_1125888_20.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/00db2054-d53b-4313-9f38-4bffe912ed86_isrForCrosstalkSources_1125888_20.stderr parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash e15ab56cc54ea2f74cb7a6b0f1188d85 parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/93bcd3b2-a189-4b7b-8ac3-138fc3164a1f_isrForCrosstalkSources_1125937_30.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/93bcd3b2-a189-4b7b-8ac3-138fc3164a1f_isrForCrosstalkSources_1125937_30.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash c05bf56b6f4b506b1eaf81199a9b196d parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/18b24264-0d32-4cde-8f49-6eafba851f25_isrForCrosstalkSources_1125905_34.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/18b24264-0d32-4cde-8f49-6eafba851f25_isrForCrosstalkSources_1125905_34.stderr parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 743 has memoization hash 6b88bfad8e7918304a6c2782bc8e60ae parsl.dataflow.memoization INFO: Task 743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/ab44e016-3097-4c46-a666-6b5f2279c3c8_isrForCrosstalkSources_1125888_35.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/ab44e016-3097-4c46-a666-6b5f2279c3c8_isrForCrosstalkSources_1125888_35.stderr parsl.dataflow.dflow DEBUG: Task 744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 744 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 744 has memoization hash 39c91d9f4810fdf424d05210e9c57b83 parsl.dataflow.memoization INFO: Task 744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 744 try 0 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/a2eff311-13e3-4f23-920b-55f78e238ce7_isrForCrosstalkSources_1125911_45.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/a2eff311-13e3-4f23-920b-55f78e238ce7_isrForCrosstalkSources_1125911_45.stderr parsl.dataflow.dflow DEBUG: Task 745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 745 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 745 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 745 has memoization hash 4ed8a5c888bde8b766d1d16552b71174 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.memoization INFO: Task 745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 745 try 0 launched on executor multi with executor id 746 parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/e8a2d553-839a-49b0-be2c-308b670bd86e_isrForCrosstalkSources_1125893_60.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/e8a2d553-839a-49b0-be2c-308b670bd86e_isrForCrosstalkSources_1125893_60.stderr parsl.dataflow.dflow DEBUG: Task 746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 746 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 746 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 746 has memoization hash 924161e4fbf49cf30c259bb9c16443b5 parsl.dataflow.memoization INFO: Task 746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 746 try 0 launched on executor multi with executor id 747 parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/5046da32-6f73-44f4-9eac-95895e81de76_isrForCrosstalkSources_1125899_44.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/5046da32-6f73-44f4-9eac-95895e81de76_isrForCrosstalkSources_1125899_44.stderr parsl.dataflow.dflow DEBUG: Task 747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 747 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 747 has memoization hash 2ebb39eae9293369ed1ce2fbb2528162 parsl.dataflow.memoization INFO: Task 747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 747 try 0 launched on executor multi with executor id 748 parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/0a7d360e-eedb-4dde-beed-00931e1e3737_isrForCrosstalkSources_1125943_8.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/0a7d360e-eedb-4dde-beed-00931e1e3737_isrForCrosstalkSources_1125943_8.stderr parsl.dataflow.dflow DEBUG: Task 748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 748 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 748 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 748 has memoization hash f6a67a854e697127dd67d4a94f6d7ea6 parsl.dataflow.memoization INFO: Task 748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 748 try 0 launched on executor multi with executor id 749 parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/c45aae5a-1318-4cb2-8e87-7b0b00c1806d_isrForCrosstalkSources_1125892_38.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/c45aae5a-1318-4cb2-8e87-7b0b00c1806d_isrForCrosstalkSources_1125892_38.stderr parsl.dataflow.dflow DEBUG: Task 749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 749 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 749 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 749 has memoization hash e5161517c4550ac4a66f645f0032810b parsl.dataflow.memoization INFO: Task 749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 749 try 0 launched on executor multi with executor id 750 parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/e0869e3f-0eaf-47af-8b4d-b1ad4332c87f_isrForCrosstalkSources_1125912_59.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/e0869e3f-0eaf-47af-8b4d-b1ad4332c87f_isrForCrosstalkSources_1125912_59.stderr parsl.dataflow.dflow DEBUG: Task 750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 750 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 750 has memoization hash 2b1dfc5ca7e82e58a84df5172201d812 parsl.dataflow.memoization INFO: Task 750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 750 try 0 launched on executor multi with executor id 751 parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/e4cd3189-80cf-4588-ad23-b5137124b9b7_isrForCrosstalkSources_1125896_52.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/e4cd3189-80cf-4588-ad23-b5137124b9b7_isrForCrosstalkSources_1125896_52.stderr parsl.dataflow.dflow DEBUG: Task 751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 751 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 751 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 751 has memoization hash 0d9ff323c1df5083b03b9b16f656a35e parsl.dataflow.memoization INFO: Task 751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 751 try 0 launched on executor multi with executor id 752 parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/e0d3571a-e7cf-4f32-bad2-87516f494c69_isrForCrosstalkSources_1125900_40.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/e0d3571a-e7cf-4f32-bad2-87516f494c69_isrForCrosstalkSources_1125900_40.stderr parsl.dataflow.dflow DEBUG: Task 752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 752 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 752 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 752 has memoization hash 9daf9d2b112d09aedfb6af93dea15500 parsl.dataflow.memoization INFO: Task 752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 752 try 0 launched on executor multi with executor id 753 parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/564dd0c4-21fe-45ef-b8f3-a2c5747796ae_isrForCrosstalkSources_1125931_31.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/564dd0c4-21fe-45ef-b8f3-a2c5747796ae_isrForCrosstalkSources_1125931_31.stderr parsl.dataflow.dflow DEBUG: Task 753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 753 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 753 has memoization hash 27028a6b6dd652dc940a7e807037bb3e parsl.dataflow.memoization INFO: Task 753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 753 try 0 launched on executor multi with executor id 754 parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d87e8e02-e2c9-4d91-b4b3-4cb40178597f_isrForCrosstalkSources_1125940_7.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d87e8e02-e2c9-4d91-b4b3-4cb40178597f_isrForCrosstalkSources_1125940_7.stderr parsl.dataflow.dflow DEBUG: Task 754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 754 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 754 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 754 has memoization hash 198f742486ca77ee7e33ba311260816b parsl.dataflow.memoization INFO: Task 754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 754 try 0 launched on executor multi with executor id 755 parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/c228f2b6-1e61-434f-acab-783f0e832b94_isrForCrosstalkSources_1125941_37.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/c228f2b6-1e61-434f-acab-783f0e832b94_isrForCrosstalkSources_1125941_37.stderr parsl.dataflow.dflow DEBUG: Task 755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 755 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 755 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 755 has memoization hash 7062f7176890e557b034f372585bd5f4 parsl.dataflow.memoization INFO: Task 755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 755 try 0 launched on executor multi with executor id 756 parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/53dd4eca-47ed-4bef-b362-0db4c00e86b4_isrForCrosstalkSources_1125895_40.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/53dd4eca-47ed-4bef-b362-0db4c00e86b4_isrForCrosstalkSources_1125895_40.stderr parsl.dataflow.dflow DEBUG: Task 756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 756 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 756 has memoization hash d7638f71a69e956b98efe6ca699731ea parsl.dataflow.memoization INFO: Task 756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 756 try 0 launched on executor multi with executor id 757 parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/9d470e4e-af5b-49d6-b982-6c32f290376c_isrForCrosstalkSources_1125944_59.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/9d470e4e-af5b-49d6-b982-6c32f290376c_isrForCrosstalkSources_1125944_59.stderr parsl.dataflow.dflow DEBUG: Task 757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 757 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 757 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 757 has memoization hash a4211dd26733daca9e031b9d40aca8f6 parsl.dataflow.memoization INFO: Task 757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 757 try 0 launched on executor multi with executor id 758 parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/a34518d4-0031-45d7-902d-f93ad2a2776a_isrForCrosstalkSources_1125927_48.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/a34518d4-0031-45d7-902d-f93ad2a2776a_isrForCrosstalkSources_1125927_48.stderr parsl.dataflow.dflow DEBUG: Task 758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 758 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 758 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 758 has memoization hash 0ab3317277df8bde0135c48c0cf96293 parsl.dataflow.memoization INFO: Task 758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 758 try 0 launched on executor multi with executor id 759 parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/e771613c-402a-420d-8bc1-ef52922195a9_isrForCrosstalkSources_1125888_50.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/e771613c-402a-420d-8bc1-ef52922195a9_isrForCrosstalkSources_1125888_50.stderr parsl.dataflow.dflow DEBUG: Task 759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 759 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 759 has memoization hash 3fb28f9c67d71eb4a399930aff59de95 parsl.dataflow.memoization INFO: Task 759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 759 try 0 launched on executor multi with executor id 760 parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/7225b83b-d58c-4c0d-80d5-f67808fb65ca_isrForCrosstalkSources_1125912_41.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/7225b83b-d58c-4c0d-80d5-f67808fb65ca_isrForCrosstalkSources_1125912_41.stderr parsl.dataflow.dflow DEBUG: Task 760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 760 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 760 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 760 has memoization hash 3b8600de244ad9cf11d84b7fe5510b50 parsl.dataflow.memoization INFO: Task 760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 760 try 0 launched on executor multi with executor id 761 parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ba036c59-2567-48ee-805b-1a2b054cbeb5_isrForCrosstalkSources_1125918_61.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ba036c59-2567-48ee-805b-1a2b054cbeb5_isrForCrosstalkSources_1125918_61.stderr parsl.dataflow.dflow DEBUG: Task 761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 761 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 761 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 761 has memoization hash c6f8e639d2046867bca28d19e88f2e0b parsl.dataflow.memoization INFO: Task 761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 761 try 0 launched on executor multi with executor id 762 parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/b49db1c5-20eb-40ff-a001-23e05717be2e_isrForCrosstalkSources_1125895_55.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/b49db1c5-20eb-40ff-a001-23e05717be2e_isrForCrosstalkSources_1125895_55.stderr parsl.dataflow.dflow DEBUG: Task 762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 762 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 762 has memoization hash c4118fae4ca942208a2016a615248205 parsl.dataflow.memoization INFO: Task 762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:42 proc_lsst.multi:170 [INFO] [multi] job_id 16896390 proc_lsst.multi INFO: [multi] job_id 16896390 2024-03-07 15:06:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 15:06:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896390 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896390 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16896390 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-07 15:06:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-07 15:06:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 15:06:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 762 try 0 launched on executor multi with executor id 763 parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/a7e3785c-554a-4877-94b0-9b5d08428278_isrForCrosstalkSources_1125920_21.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/a7e3785c-554a-4877-94b0-9b5d08428278_isrForCrosstalkSources_1125920_21.stderr parsl.dataflow.dflow DEBUG: Task 763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 763 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 763 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 763 has memoization hash 00f7c116433a9b2dcb662a6f3c84ee5c parsl.dataflow.memoization INFO: Task 763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d267740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 763 try 0 launched on executor multi with executor id 764 parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/0cdab88b-bbc3-4596-9043-fedccd6399a7_isrForCrosstalkSources_1125909_33.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/0cdab88b-bbc3-4596-9043-fedccd6399a7_isrForCrosstalkSources_1125909_33.stderr parsl.dataflow.dflow DEBUG: Task 764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 764 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 764 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 764 has memoization hash 10bf248801c70bf8ae2c10fe49c97224 parsl.dataflow.memoization INFO: Task 764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3ce2e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 764 try 0 launched on executor multi with executor id 765 parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/9fd9e7e4-b70f-4e98-a0f1-4cea85849174_isrForCrosstalkSources_1125905_32.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/9fd9e7e4-b70f-4e98-a0f1-4cea85849174_isrForCrosstalkSources_1125905_32.stderr parsl.dataflow.dflow DEBUG: Task 765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 765 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 765 has memoization hash 68d26701620e357b6218df073df33ef6 parsl.dataflow.memoization INFO: Task 765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 765 try 0 launched on executor multi with executor id 766 parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/c048a176-5cf1-42c8-a614-47a7ba092fec_isrForCrosstalkSources_1125899_25.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/c048a176-5cf1-42c8-a614-47a7ba092fec_isrForCrosstalkSources_1125899_25.stderr parsl.dataflow.dflow DEBUG: Task 766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 766 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 766 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 766 has memoization hash 15944d1f86061c1e40158b0e562bbb9e parsl.dataflow.memoization INFO: Task 766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 766 try 0 launched on executor multi with executor id 767 parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/5e10e291-c105-4c88-aed5-659bf66a6205_isrForCrosstalkSources_1125890_20.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/5e10e291-c105-4c88-aed5-659bf66a6205_isrForCrosstalkSources_1125890_20.stderr parsl.dataflow.dflow DEBUG: Task 767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 767 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 767 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 767 has memoization hash 6349a01f13dfc7cbb39a05472515a680 parsl.dataflow.memoization INFO: Task 767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 767 try 0 launched on executor multi with executor id 768 parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/0801ea30-7ee2-4d21-bb73-632e152f5d54_isrForCrosstalkSources_1125895_20.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/0801ea30-7ee2-4d21-bb73-632e152f5d54_isrForCrosstalkSources_1125895_20.stderr parsl.dataflow.dflow DEBUG: Task 768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 768 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 768 has memoization hash b5973c85167b0e972dcdd568813a6953 parsl.dataflow.memoization INFO: Task 768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 768 try 0 launched on executor multi with executor id 769 parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/8c38477a-702b-4446-8aaf-34adc11e640f_isrForCrosstalkSources_1125923_54.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/8c38477a-702b-4446-8aaf-34adc11e640f_isrForCrosstalkSources_1125923_54.stderr parsl.dataflow.dflow DEBUG: Task 769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 769 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 769 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 769 has memoization hash 083697699cdcae61bf2d205eba8c6988 parsl.dataflow.memoization INFO: Task 769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 769 try 0 launched on executor multi with executor id 770 parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/5beec3d1-1034-44e1-b23c-e153bd809f42_isrForCrosstalkSources_1125930_8.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/5beec3d1-1034-44e1-b23c-e153bd809f42_isrForCrosstalkSources_1125930_8.stderr parsl.dataflow.dflow DEBUG: Task 770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 770 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 770 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 770 has memoization hash eac7c6b6e88901919fc8656cde8fcbc0 parsl.dataflow.memoization INFO: Task 770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 770 try 0 launched on executor multi with executor id 771 parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/86a1c3b6-d12d-4c1b-9965-8164be7e256f_isrForCrosstalkSources_1125924_4.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/86a1c3b6-d12d-4c1b-9965-8164be7e256f_isrForCrosstalkSources_1125924_4.stderr parsl.dataflow.dflow DEBUG: Task 771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 771 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 771 has memoization hash 6d0b72fc8e53654b89d6057bf04290b5 parsl.dataflow.memoization INFO: Task 771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 771 try 0 launched on executor multi with executor id 772 parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/ebc09346-f981-453a-a9e8-2570f8719c5e_isrForCrosstalkSources_1125893_55.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/ebc09346-f981-453a-a9e8-2570f8719c5e_isrForCrosstalkSources_1125893_55.stderr parsl.dataflow.dflow DEBUG: Task 772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 772 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 772 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 772 has memoization hash fb512bed757b42a7ea6346988df08960 parsl.dataflow.memoization INFO: Task 772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 772 try 0 launched on executor multi with executor id 773 parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/581809aa-6fa0-4c7e-97c2-6d7d7a4e0266_isrForCrosstalkSources_1125903_24.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/581809aa-6fa0-4c7e-97c2-6d7d7a4e0266_isrForCrosstalkSources_1125903_24.stderr parsl.dataflow.dflow DEBUG: Task 773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 773 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 773 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 773 has memoization hash 07e2ea4f13d530b0288c571dc5c39a9b parsl.dataflow.memoization INFO: Task 773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 773 try 0 launched on executor multi with executor id 774 parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/319b2458-8a83-4802-9276-714335542650_isrForCrosstalkSources_1125905_57.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/319b2458-8a83-4802-9276-714335542650_isrForCrosstalkSources_1125905_57.stderr parsl.dataflow.dflow DEBUG: Task 774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 774 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 774 has memoization hash 27d38b1c01fb038dd1c6f38c79236d26 parsl.dataflow.memoization INFO: Task 774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 774 try 0 launched on executor multi with executor id 775 parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/ef1459ce-86da-4d42-9095-907424e3dc31_isrForCrosstalkSources_1125928_20.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/ef1459ce-86da-4d42-9095-907424e3dc31_isrForCrosstalkSources_1125928_20.stderr parsl.dataflow.dflow DEBUG: Task 775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 775 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 775 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 775 has memoization hash 495dff810df904fcf8adf42e421f1eb2 parsl.dataflow.memoization INFO: Task 775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 775 try 0 launched on executor multi with executor id 776 parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/1e28458e-4ee9-449f-9a15-a7dff646017e_isrForCrosstalkSources_1125899_49.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/1e28458e-4ee9-449f-9a15-a7dff646017e_isrForCrosstalkSources_1125899_49.stderr parsl.dataflow.dflow DEBUG: Task 776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 776 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 776 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 776 has memoization hash 23b827bb10dcba09414e3bb85388b3bc parsl.dataflow.memoization INFO: Task 776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 776 try 0 launched on executor multi with executor id 777 parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/d449ea95-61fd-49e1-9096-0b6899d0b744_isrForCrosstalkSources_1125895_25.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/d449ea95-61fd-49e1-9096-0b6899d0b744_isrForCrosstalkSources_1125895_25.stderr parsl.dataflow.dflow DEBUG: Task 777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 777 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 777 has memoization hash 080f91460f9be9ed46c4a9674bd7e707 parsl.dataflow.memoization INFO: Task 777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 777 try 0 launched on executor multi with executor id 778 parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/6549b3c9-195b-43ae-9e06-4e20125ea8f7_isrForCrosstalkSources_1125924_24.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/6549b3c9-195b-43ae-9e06-4e20125ea8f7_isrForCrosstalkSources_1125924_24.stderr parsl.dataflow.dflow DEBUG: Task 778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 778 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 778 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 778 has memoization hash ae0f891351ea1febb0ff14836f949f66 parsl.dataflow.memoization INFO: Task 778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 778 try 0 launched on executor multi with executor id 779 parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/85f2d6da-a658-43e6-bb8b-69091faf4731_isrForCrosstalkSources_1125942_3.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/85f2d6da-a658-43e6-bb8b-69091faf4731_isrForCrosstalkSources_1125942_3.stderr parsl.dataflow.dflow DEBUG: Task 779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 779 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 779 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 779 has memoization hash f51b208ed41ac4034d22912d9ab5fb59 parsl.dataflow.memoization INFO: Task 779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 779 try 0 launched on executor multi with executor id 780 parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1e2f2251-29d1-4f04-a944-f785977e6e5e_isrForCrosstalkSources_1125900_33.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1e2f2251-29d1-4f04-a944-f785977e6e5e_isrForCrosstalkSources_1125900_33.stderr parsl.dataflow.dflow DEBUG: Task 780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 780 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 780 has memoization hash b2b022324d9e1e82933f2ad53445f2bc parsl.dataflow.memoization INFO: Task 780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 780 try 0 launched on executor multi with executor id 781 parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/23cc97d7-c41e-428b-852e-32fdaddff229_isrForCrosstalkSources_1125888_44.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/23cc97d7-c41e-428b-852e-32fdaddff229_isrForCrosstalkSources_1125888_44.stderr parsl.dataflow.dflow DEBUG: Task 781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 781 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 781 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 781 has memoization hash 811cbf4f4ab4c370b23e72c8cccb9a49 parsl.dataflow.memoization INFO: Task 781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 781 try 0 launched on executor multi with executor id 782 parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/2b03546e-8e57-4897-92c3-443227be21cb_isrForCrosstalkSources_1125938_41.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/2b03546e-8e57-4897-92c3-443227be21cb_isrForCrosstalkSources_1125938_41.stderr parsl.dataflow.dflow DEBUG: Task 782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 782 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 782 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 782 has memoization hash 816a0bf72e8c8d5113f38784d42d7486 parsl.dataflow.memoization INFO: Task 782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 782 try 0 launched on executor multi with executor id 783 parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/7ecee4f1-393e-4218-a2b4-074fe02535ae_isrForCrosstalkSources_1125937_32.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/7ecee4f1-393e-4218-a2b4-074fe02535ae_isrForCrosstalkSources_1125937_32.stderr parsl.dataflow.dflow DEBUG: Task 783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 783 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 783 has memoization hash 920e4902f93e1c0ea02fe1c3193c8229 parsl.dataflow.memoization INFO: Task 783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 783 try 0 launched on executor multi with executor id 784 parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/52c7774f-424f-462b-8a17-b342d9d3f525_isrForCrosstalkSources_1125920_1.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/52c7774f-424f-462b-8a17-b342d9d3f525_isrForCrosstalkSources_1125920_1.stderr parsl.dataflow.dflow DEBUG: Task 784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 784 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 784 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 784 has memoization hash cc7e912dea2fd65906affb978df42437 parsl.dataflow.memoization INFO: Task 784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 784 try 0 launched on executor multi with executor id 785 parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4cba92f3-b216-49b8-be84-fc391c3758bd_isrForCrosstalkSources_1125893_27.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4cba92f3-b216-49b8-be84-fc391c3758bd_isrForCrosstalkSources_1125893_27.stderr parsl.dataflow.dflow DEBUG: Task 785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 785 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 785 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 785 has memoization hash 33bccc1dd4a7f4f5cdfccdb162a7feda parsl.dataflow.memoization INFO: Task 785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 785 try 0 launched on executor multi with executor id 786 parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/2534042c-26ed-4321-bbc4-9cf9cb3d8d81_isrForCrosstalkSources_1125918_30.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/2534042c-26ed-4321-bbc4-9cf9cb3d8d81_isrForCrosstalkSources_1125918_30.stderr parsl.dataflow.dflow DEBUG: Task 786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 786 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 786 has memoization hash 4c93efa40a39ca7f7cd468a647c41ff7 parsl.dataflow.memoization INFO: Task 786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 786 try 0 launched on executor multi with executor id 787 parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/412f19b0-36aa-474a-8d8b-4ee0e8b02c35_isrForCrosstalkSources_1125917_40.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/412f19b0-36aa-474a-8d8b-4ee0e8b02c35_isrForCrosstalkSources_1125917_40.stderr parsl.dataflow.dflow DEBUG: Task 787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 787 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 787 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 787 has memoization hash 19abdfb9d42501c6a990e3095500bff4 parsl.dataflow.memoization INFO: Task 787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 787 try 0 launched on executor multi with executor id 788 parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/b7468187-cc4b-43da-8f7d-7545acf42797_isrForCrosstalkSources_1125917_53.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/b7468187-cc4b-43da-8f7d-7545acf42797_isrForCrosstalkSources_1125917_53.stderr parsl.dataflow.dflow DEBUG: Task 788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 788 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 788 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 788 has memoization hash 1fed8530ed3c9ef0b0a80d1d8b4a697f parsl.dataflow.memoization INFO: Task 788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 788 try 0 launched on executor multi with executor id 789 parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/d57e8ffc-502a-4533-97a3-414dbba8dcca_isrForCrosstalkSources_1125892_11.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/d57e8ffc-502a-4533-97a3-414dbba8dcca_isrForCrosstalkSources_1125892_11.stderr parsl.dataflow.dflow DEBUG: Task 789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 789 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 789 has memoization hash a5ab132ab754afaa45614d6fa70ebb25 parsl.dataflow.memoization INFO: Task 789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 789 try 0 launched on executor multi with executor id 790 parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/61ed8451-e7db-4c08-a466-ef985182e642_isrForCrosstalkSources_1125932_4.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/61ed8451-e7db-4c08-a466-ef985182e642_isrForCrosstalkSources_1125932_4.stderr parsl.dataflow.dflow DEBUG: Task 790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 790 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 790 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 790 has memoization hash 3047a1a7549962f027d497b4e8a9bb55 parsl.dataflow.memoization INFO: Task 790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 790 try 0 launched on executor multi with executor id 791 parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/77d92217-e7ac-4fc3-a8bf-8bae61360866_isrForCrosstalkSources_1125934_43.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/77d92217-e7ac-4fc3-a8bf-8bae61360866_isrForCrosstalkSources_1125934_43.stderr parsl.dataflow.dflow DEBUG: Task 791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 791 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 791 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 791 has memoization hash 16727a4626554f4c892588ec7e2624e4 parsl.dataflow.memoization INFO: Task 791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 791 try 0 launched on executor multi with executor id 792 parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/d4bc819f-861c-4ab4-a68e-49db0daf70cd_isrForCrosstalkSources_1125926_33.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/d4bc819f-861c-4ab4-a68e-49db0daf70cd_isrForCrosstalkSources_1125926_33.stderr parsl.dataflow.dflow DEBUG: Task 792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 792 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 792 has memoization hash d4a0ca3550d96ed477e05ad59027117b parsl.dataflow.memoization INFO: Task 792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 792 try 0 launched on executor multi with executor id 793 parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/0a38e9a0-8f5f-4ea2-91b8-396117077190_isrForCrosstalkSources_1125909_27.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/0a38e9a0-8f5f-4ea2-91b8-396117077190_isrForCrosstalkSources_1125909_27.stderr parsl.dataflow.dflow DEBUG: Task 793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 793 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 793 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 793 has memoization hash e0b1dc2feb1f8d82e9f0bd79011c4f14 parsl.dataflow.memoization INFO: Task 793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 793 try 0 launched on executor multi with executor id 794 parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/79a07bba-4a6c-449a-9c06-bb27ba63c1bb_isrForCrosstalkSources_1125924_58.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/79a07bba-4a6c-449a-9c06-bb27ba63c1bb_isrForCrosstalkSources_1125924_58.stderr parsl.dataflow.dflow DEBUG: Task 794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 794 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 794 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 794 has memoization hash 3e3951d7dad97b8145a4565cd7df9a72 parsl.dataflow.memoization INFO: Task 794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 794 try 0 launched on executor multi with executor id 795 parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f4729d3a-cfc0-4c54-a820-3371cbca83cc_isrForCrosstalkSources_1125894_40.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f4729d3a-cfc0-4c54-a820-3371cbca83cc_isrForCrosstalkSources_1125894_40.stderr parsl.dataflow.dflow DEBUG: Task 795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 795 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 795 has memoization hash 9a7d00c4980406c9fb98f75fc34443b7 parsl.dataflow.memoization INFO: Task 795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 795 try 0 launched on executor multi with executor id 796 parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/773e047f-ca47-412e-871c-492d7f06dd1e_isrForCrosstalkSources_1125904_60.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/773e047f-ca47-412e-871c-492d7f06dd1e_isrForCrosstalkSources_1125904_60.stderr parsl.dataflow.dflow DEBUG: Task 796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 796 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 796 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 796 has memoization hash d0d1dfcacb4f4ee520a4c03f6eb9ca7f parsl.dataflow.memoization INFO: Task 796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 796 try 0 launched on executor multi with executor id 797 parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/83bd98f5-eefd-4959-9de6-d969047d5bc9_isrForCrosstalkSources_1125944_52.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/83bd98f5-eefd-4959-9de6-d969047d5bc9_isrForCrosstalkSources_1125944_52.stderr parsl.dataflow.dflow DEBUG: Task 797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 797 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 797 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 797 has memoization hash 301041647043c40f2939a0027f13a8ea parsl.dataflow.memoization INFO: Task 797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 797 try 0 launched on executor multi with executor id 798 parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/825162ce-075f-4c34-bdbc-30e6c0b224f2_isrForCrosstalkSources_1125941_32.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/825162ce-075f-4c34-bdbc-30e6c0b224f2_isrForCrosstalkSources_1125941_32.stderr parsl.dataflow.dflow DEBUG: Task 798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 798 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 798 has memoization hash 3d23fe99f20855c1024ffe3cce74be5b parsl.dataflow.memoization INFO: Task 798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 798 try 0 launched on executor multi with executor id 799 parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/a3d8521a-aa0d-4ae6-b4e0-b5b805e9cf5c_isrForCrosstalkSources_1125924_7.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/a3d8521a-aa0d-4ae6-b4e0-b5b805e9cf5c_isrForCrosstalkSources_1125924_7.stderr parsl.dataflow.dflow DEBUG: Task 799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 799 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 799 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 799 has memoization hash b47ec116e1a4a362138bed0485c5d569 parsl.dataflow.memoization INFO: Task 799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 799 try 0 launched on executor multi with executor id 800 parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/7f7142b2-aebb-4534-9336-277b00fe3cf7_isrForCrosstalkSources_1125900_23.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/7f7142b2-aebb-4534-9336-277b00fe3cf7_isrForCrosstalkSources_1125900_23.stderr parsl.dataflow.dflow DEBUG: Task 800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 800 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 800 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 800 has memoization hash 972ebd609efe0ed3a999401555f9da0c parsl.dataflow.memoization INFO: Task 800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 800 try 0 launched on executor multi with executor id 801 parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/8d149064-3467-476c-9c6b-c882bc94eb36_isrForCrosstalkSources_1125912_47.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/8d149064-3467-476c-9c6b-c882bc94eb36_isrForCrosstalkSources_1125912_47.stderr parsl.dataflow.dflow DEBUG: Task 801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 801 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 801 has memoization hash ddb85a378ba3869dba317f054ccc1283 parsl.dataflow.memoization INFO: Task 801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:43 proc_lsst.multi:170 [INFO] [multi] job_id 16896392 proc_lsst.multi INFO: [multi] job_id 16896392 2024-03-07 15:06:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 15:06:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896392 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896392 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16896392 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-07 15:06:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-07 15:06:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 15:06:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 801 try 0 launched on executor multi with executor id 802 parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/c5e2dae4-f0b3-436b-a61f-da8fabf02b31_isrForCrosstalkSources_1125946_56.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/c5e2dae4-f0b3-436b-a61f-da8fabf02b31_isrForCrosstalkSources_1125946_56.stderr parsl.dataflow.dflow DEBUG: Task 802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 802 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 802 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 802 has memoization hash 1659f5473aa9799b65eff96202c0eb1a parsl.dataflow.memoization INFO: Task 802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 802 try 0 launched on executor multi with executor id 803 parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d82c489e-3205-4ae7-b92e-1c2f6ce14f4c_isrForCrosstalkSources_1125934_30.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d82c489e-3205-4ae7-b92e-1c2f6ce14f4c_isrForCrosstalkSources_1125934_30.stderr parsl.dataflow.dflow DEBUG: Task 803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 803 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 803 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 803 has memoization hash 7fe40f235aefb1d3aba13b27b6c3f3c8 parsl.dataflow.memoization INFO: Task 803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 803 try 0 launched on executor multi with executor id 804 parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/92646f71-0bbf-49b9-8d71-4be5b9b618d0_isrForCrosstalkSources_1125932_17.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/92646f71-0bbf-49b9-8d71-4be5b9b618d0_isrForCrosstalkSources_1125932_17.stderr parsl.dataflow.dflow DEBUG: Task 804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 804 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 804 has memoization hash 33f6cd1781404eef1b0f04254aab3add parsl.dataflow.memoization INFO: Task 804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 804 try 0 launched on executor multi with executor id 805 parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/86b730fe-bc65-4a6c-bcf9-6a81a1637339_isrForCrosstalkSources_1125915_37.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/86b730fe-bc65-4a6c-bcf9-6a81a1637339_isrForCrosstalkSources_1125915_37.stderr parsl.dataflow.dflow DEBUG: Task 805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 805 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 805 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 805 has memoization hash 306e2761f99aa3a04e3ad61fc2170281 parsl.dataflow.memoization INFO: Task 805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 805 try 0 launched on executor multi with executor id 806 parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/2657dc2a-de4d-44e1-a46b-d9cc7625dfaf_isrForCrosstalkSources_1125905_3.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/2657dc2a-de4d-44e1-a46b-d9cc7625dfaf_isrForCrosstalkSources_1125905_3.stderr parsl.dataflow.dflow DEBUG: Task 806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 806 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 806 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 806 has memoization hash a46978b9cb3c6b1f9a9da06522d7805e parsl.dataflow.memoization INFO: Task 806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d266200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 806 try 0 launched on executor multi with executor id 807 parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/bbc48eed-084b-4244-be11-3d7fec17b472_isrForCrosstalkSources_1125912_28.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/bbc48eed-084b-4244-be11-3d7fec17b472_isrForCrosstalkSources_1125912_28.stderr parsl.dataflow.dflow DEBUG: Task 807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 807 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 807 has memoization hash 6f7eca7ae2da22e2207de24b819f955d parsl.dataflow.memoization INFO: Task 807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 807 try 0 launched on executor multi with executor id 808 parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/16bc2415-ae65-4bd6-acbb-93d1f485c80c_isrForCrosstalkSources_1125946_10.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/16bc2415-ae65-4bd6-acbb-93d1f485c80c_isrForCrosstalkSources_1125946_10.stderr parsl.dataflow.dflow DEBUG: Task 808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 808 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 808 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 808 has memoization hash 1cb89e4e450669846a8c1a31a0ccd529 parsl.dataflow.memoization INFO: Task 808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 808 try 0 launched on executor multi with executor id 809 parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/3a94667e-48bf-4577-ab7e-74c00bf41afd_isrForCrosstalkSources_1125894_9.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/3a94667e-48bf-4577-ab7e-74c00bf41afd_isrForCrosstalkSources_1125894_9.stderr parsl.dataflow.dflow DEBUG: Task 809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 809 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 809 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 809 has memoization hash 308437c320d1f47b35d80ae45315c458 parsl.dataflow.memoization INFO: Task 809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 809 try 0 launched on executor multi with executor id 810 parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/9fd960f9-2c4e-488d-8a3e-36e3c45d2236_isrForCrosstalkSources_1125905_4.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/9fd960f9-2c4e-488d-8a3e-36e3c45d2236_isrForCrosstalkSources_1125905_4.stderr parsl.dataflow.dflow DEBUG: Task 810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 810 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 810 has memoization hash 9cc6caad8c5229a97101a8baf358a643 parsl.dataflow.memoization INFO: Task 810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 810 try 0 launched on executor multi with executor id 811 parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ef7813e9-bcc4-470f-83fb-b5d04326ce12_isrForCrosstalkSources_1125894_7.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ef7813e9-bcc4-470f-83fb-b5d04326ce12_isrForCrosstalkSources_1125894_7.stderr parsl.dataflow.dflow DEBUG: Task 811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 811 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 811 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 811 has memoization hash 08a299091c867ad5fbe76e22550eb4fc parsl.dataflow.memoization INFO: Task 811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 811 try 0 launched on executor multi with executor id 812 parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3a06442a-3b8e-4bf5-b1c5-7ca4d24f2711_isrForCrosstalkSources_1125912_31.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3a06442a-3b8e-4bf5-b1c5-7ca4d24f2711_isrForCrosstalkSources_1125912_31.stderr parsl.dataflow.dflow DEBUG: Task 812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 812 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 812 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 812 has memoization hash 56f4f20ec6068168ed3327f66bed44d3 parsl.dataflow.memoization INFO: Task 812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 812 try 0 launched on executor multi with executor id 813 parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/e94fa875-7008-4732-aa41-c1a2fbd6ccfe_isrForCrosstalkSources_1125936_1.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/e94fa875-7008-4732-aa41-c1a2fbd6ccfe_isrForCrosstalkSources_1125936_1.stderr parsl.dataflow.dflow DEBUG: Task 813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 813 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 813 has memoization hash 343314ae6ca9a31831f3d0bbd0a10dc7 parsl.dataflow.memoization INFO: Task 813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 813 try 0 launched on executor multi with executor id 814 parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/190aee37-d5fe-4de4-a860-413ccbf617a6_isrForCrosstalkSources_1125937_25.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/190aee37-d5fe-4de4-a860-413ccbf617a6_isrForCrosstalkSources_1125937_25.stderr parsl.dataflow.dflow DEBUG: Task 814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 814 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 814 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 814 has memoization hash ac5951f85c10dacff0a6a1bf6643470e parsl.dataflow.memoization INFO: Task 814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 814 try 0 launched on executor multi with executor id 815 parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/64ae782f-c567-42cb-b876-5e6cee337063_isrForCrosstalkSources_1125888_36.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/64ae782f-c567-42cb-b876-5e6cee337063_isrForCrosstalkSources_1125888_36.stderr parsl.dataflow.dflow DEBUG: Task 815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 815 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 815 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 815 has memoization hash a7989c962b275bbcd26b331f08001c8a parsl.dataflow.memoization INFO: Task 815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 815 try 0 launched on executor multi with executor id 816 parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/2898608d-51a1-4fc8-9243-56a9c6b4e88b_isrForCrosstalkSources_1125918_58.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/2898608d-51a1-4fc8-9243-56a9c6b4e88b_isrForCrosstalkSources_1125918_58.stderr parsl.dataflow.dflow DEBUG: Task 816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 816 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 816 has memoization hash cea686959ab58bf83748a0c89c00a82e parsl.dataflow.memoization INFO: Task 816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 816 try 0 launched on executor multi with executor id 817 parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/c12b7b77-c92c-4683-ac3c-82f63beff290_isrForCrosstalkSources_1125911_10.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/c12b7b77-c92c-4683-ac3c-82f63beff290_isrForCrosstalkSources_1125911_10.stderr parsl.dataflow.dflow DEBUG: Task 817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 817 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 817 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 817 has memoization hash a8de203b7a4ae43db051928711c5551c parsl.dataflow.memoization INFO: Task 817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 817 try 0 launched on executor multi with executor id 818 parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/d402793e-3a8d-4799-9a7d-91b55bed49df_isrForCrosstalkSources_1125901_13.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/d402793e-3a8d-4799-9a7d-91b55bed49df_isrForCrosstalkSources_1125901_13.stderr parsl.dataflow.dflow DEBUG: Task 818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 818 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 818 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 818 has memoization hash 5e57fa057a505038357b76074321c5c8 parsl.dataflow.memoization INFO: Task 818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 818 try 0 launched on executor multi with executor id 819 parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/b6daafd2-0149-4d61-a62d-9ed1c18ebba7_isrForCrosstalkSources_1125902_42.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/b6daafd2-0149-4d61-a62d-9ed1c18ebba7_isrForCrosstalkSources_1125902_42.stderr parsl.dataflow.dflow DEBUG: Task 819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 819 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 819 has memoization hash af2e34c4840f9ade07574e23cc1fef31 parsl.dataflow.memoization INFO: Task 819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 819 try 0 launched on executor multi with executor id 820 parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/10166b6a-856a-4fe0-9e36-68fb54f8e7da_isrForCrosstalkSources_1125890_9.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/10166b6a-856a-4fe0-9e36-68fb54f8e7da_isrForCrosstalkSources_1125890_9.stderr parsl.dataflow.dflow DEBUG: Task 820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 820 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 820 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 820 has memoization hash 2daf8cefc64e9be2f7ab8e7fe7aa9840 parsl.dataflow.memoization INFO: Task 820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 820 try 0 launched on executor multi with executor id 821 parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/571c35f3-af38-449c-8aea-69b37b40e1e9_isrForCrosstalkSources_1125895_32.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/571c35f3-af38-449c-8aea-69b37b40e1e9_isrForCrosstalkSources_1125895_32.stderr 2024-03-07 15:06:43 proc_lsst.multi:170 [INFO] [multi] job_id 16896394 proc_lsst.multi INFO: [multi] job_id 16896394 2024-03-07 15:06:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 15:06:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896394 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896394 parsl.dataflow.dflow DEBUG: Task 821 will be sent to executor multi parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16896394 parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.executors.status_handling INFO: Allocated block ID 9 parsl.dataflow.dflow INFO: Task 821 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-07 15:06:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-07 15:06:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 15:06:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.dataflow.dflow DEBUG: Task 821 set to pending state with AppFuture: parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 821 has memoization hash 4968bfc9049bc91f58dbcceff815da46 parsl.dataflow.memoization INFO: Task 821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 821 try 0 launched on executor multi with executor id 822 parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e46dbcf7-a3d7-4aa1-95ff-72b305d1919d_isrForCrosstalkSources_1125937_53.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e46dbcf7-a3d7-4aa1-95ff-72b305d1919d_isrForCrosstalkSources_1125937_53.stderr parsl.dataflow.dflow DEBUG: Task 822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 822 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 822 has memoization hash d0b3c18d431c6fc15780fc25eef08cca parsl.dataflow.memoization INFO: Task 822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 822 try 0 launched on executor multi with executor id 823 parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/821319cb-fbfb-4465-aa42-48667d05a398_isrForCrosstalkSources_1125908_27.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/821319cb-fbfb-4465-aa42-48667d05a398_isrForCrosstalkSources_1125908_27.stderr parsl.dataflow.dflow DEBUG: Task 823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 823 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 823 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 823 has memoization hash 8a5a612325ce4964881f92126d0c7e36 parsl.dataflow.memoization INFO: Task 823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 823 try 0 launched on executor multi with executor id 824 parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/2d67f7e7-919a-448e-beb2-b5875202a26b_isrForCrosstalkSources_1125901_27.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/2d67f7e7-919a-448e-beb2-b5875202a26b_isrForCrosstalkSources_1125901_27.stderr parsl.dataflow.dflow DEBUG: Task 824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 824 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 824 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 824 has memoization hash b0b9be01fa400be6212cf44575d10964 parsl.dataflow.memoization INFO: Task 824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 824 try 0 launched on executor multi with executor id 825 parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/d6c99096-fdf8-4576-9418-b8a307267b04_isrForCrosstalkSources_1125916_51.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/d6c99096-fdf8-4576-9418-b8a307267b04_isrForCrosstalkSources_1125916_51.stderr parsl.dataflow.dflow DEBUG: Task 825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 825 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 825 has memoization hash cf44a789fb316cc5ec510209c407933c parsl.dataflow.memoization INFO: Task 825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 825 try 0 launched on executor multi with executor id 826 parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/3885782b-2de3-42cd-808f-ccfd58185076_isrForCrosstalkSources_1125918_40.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/3885782b-2de3-42cd-808f-ccfd58185076_isrForCrosstalkSources_1125918_40.stderr parsl.dataflow.dflow DEBUG: Task 826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 826 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 826 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 826 has memoization hash 17bc8532ef4a407b15b8f4fc967d91aa parsl.dataflow.memoization INFO: Task 826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 826 try 0 launched on executor multi with executor id 827 parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/12636828-5316-48b5-b4c9-24f1371e413e_isrForCrosstalkSources_1125889_39.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/12636828-5316-48b5-b4c9-24f1371e413e_isrForCrosstalkSources_1125889_39.stderr parsl.dataflow.dflow DEBUG: Task 827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 827 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 827 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 827 has memoization hash 972f29402c8ae880735b6627af2f5208 parsl.dataflow.memoization INFO: Task 827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 827 try 0 launched on executor multi with executor id 828 parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/67110e27-ea35-416c-83ad-9a1f6c2dfb77_isrForCrosstalkSources_1125925_51.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/67110e27-ea35-416c-83ad-9a1f6c2dfb77_isrForCrosstalkSources_1125925_51.stderr parsl.dataflow.dflow DEBUG: Task 828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 828 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 828 has memoization hash 3a345dbd31f3df60711483d2b4b38dc3 parsl.dataflow.memoization INFO: Task 828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be380e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 828 try 0 launched on executor multi with executor id 829 parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/b674b963-912b-4198-9651-a1f63c941a03_isrForCrosstalkSources_1125944_16.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/b674b963-912b-4198-9651-a1f63c941a03_isrForCrosstalkSources_1125944_16.stderr parsl.dataflow.dflow DEBUG: Task 829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 829 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 829 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 829 has memoization hash ab714c8e3fd82370de2a9aaa3ece0204 parsl.dataflow.memoization INFO: Task 829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 829 try 0 launched on executor multi with executor id 830 parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/db8ceb9a-6375-423d-8873-0bf4003ef4d5_isrForCrosstalkSources_1125924_27.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/db8ceb9a-6375-423d-8873-0bf4003ef4d5_isrForCrosstalkSources_1125924_27.stderr parsl.dataflow.dflow DEBUG: Task 830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 830 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 830 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 830 has memoization hash 8c12c2648e87f236106a17c6e3b85922 parsl.dataflow.memoization INFO: Task 830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 830 try 0 launched on executor multi with executor id 831 parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/f76b9767-7209-4ab9-b34b-98aed6fd50ed_isrForCrosstalkSources_1125924_29.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/f76b9767-7209-4ab9-b34b-98aed6fd50ed_isrForCrosstalkSources_1125924_29.stderr parsl.dataflow.dflow DEBUG: Task 831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 831 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 831 has memoization hash fb8472abfc7757e5cf6d93a52a3f52c7 parsl.dataflow.memoization INFO: Task 831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 831 try 0 launched on executor multi with executor id 832 parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/e94ee1d3-988c-42b5-b821-bfed04429114_isrForCrosstalkSources_1125914_37.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/e94ee1d3-988c-42b5-b821-bfed04429114_isrForCrosstalkSources_1125914_37.stderr parsl.dataflow.dflow DEBUG: Task 832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 832 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 832 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 832 has memoization hash 2dbd00db6f8414773fd01a540f71d108 parsl.dataflow.memoization INFO: Task 832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be382c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 832 try 0 launched on executor multi with executor id 833 parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/520d3928-8750-4307-8c07-6752e3516eaf_isrForCrosstalkSources_1125931_54.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/520d3928-8750-4307-8c07-6752e3516eaf_isrForCrosstalkSources_1125931_54.stderr parsl.dataflow.dflow DEBUG: Task 833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 833 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 833 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 833 has memoization hash ffed6053a5c07f3f41a566ff7f97deb7 parsl.dataflow.memoization INFO: Task 833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 833 try 0 launched on executor multi with executor id 834 parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/7eb08dfd-e7d5-45c1-837a-0cff078bd8dd_isrForCrosstalkSources_1125899_14.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/7eb08dfd-e7d5-45c1-837a-0cff078bd8dd_isrForCrosstalkSources_1125899_14.stderr parsl.dataflow.dflow DEBUG: Task 834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 834 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 834 has memoization hash ca88877b138d330535b0928d82faefb9 parsl.dataflow.memoization INFO: Task 834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be385e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 834 try 0 launched on executor multi with executor id 835 parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/18fbfb6e-88b7-4f4b-bdbb-3d9554862ca4_isrForCrosstalkSources_1125928_40.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/18fbfb6e-88b7-4f4b-bdbb-3d9554862ca4_isrForCrosstalkSources_1125928_40.stderr parsl.dataflow.dflow DEBUG: Task 835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 835 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 835 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 835 has memoization hash bf72c186dca04c5179bf6a4f0faf2c9b parsl.dataflow.memoization INFO: Task 835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be384a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 835 try 0 launched on executor multi with executor id 836 parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/b50d5fc8-6146-4dd3-8144-4a1b1b11383f_isrForCrosstalkSources_1125945_52.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/b50d5fc8-6146-4dd3-8144-4a1b1b11383f_isrForCrosstalkSources_1125945_52.stderr parsl.dataflow.dflow DEBUG: Task 836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 836 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 836 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 836 has memoization hash be31f5bbd65b5ab5f1c16d6a585a25d7 parsl.dataflow.memoization INFO: Task 836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 836 try 0 launched on executor multi with executor id 837 parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/9399e372-5168-491b-960b-167310281e07_isrForCrosstalkSources_1125937_5.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/9399e372-5168-491b-960b-167310281e07_isrForCrosstalkSources_1125937_5.stderr parsl.dataflow.dflow DEBUG: Task 837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 837 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 837 has memoization hash 276469cf98d9a07a7ed68076d7e2c39d parsl.dataflow.memoization INFO: Task 837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be387c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 837 try 0 launched on executor multi with executor id 838 parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/7a02a875-4fcb-497b-9ec0-558708e9fb11_isrForCrosstalkSources_1125899_23.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/7a02a875-4fcb-497b-9ec0-558708e9fb11_isrForCrosstalkSources_1125899_23.stderr parsl.dataflow.dflow DEBUG: Task 838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 838 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 838 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 838 has memoization hash c16147833c378c5fdcdceb4abbb73a77 parsl.dataflow.memoization INFO: Task 838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 838 try 0 launched on executor multi with executor id 839 parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/81299053-63fe-4e7f-aebf-4ec27ec112f1_isrForCrosstalkSources_1125937_7.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/81299053-63fe-4e7f-aebf-4ec27ec112f1_isrForCrosstalkSources_1125937_7.stderr parsl.dataflow.dflow DEBUG: Task 839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 839 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 839 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 839 has memoization hash e3fed2711830e333c85f3fb5b8039536 parsl.dataflow.memoization INFO: Task 839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 839 try 0 launched on executor multi with executor id 840 parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e9efc920-6bd0-475c-96c1-5aa43e25ebf1_isrForCrosstalkSources_1125901_9.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e9efc920-6bd0-475c-96c1-5aa43e25ebf1_isrForCrosstalkSources_1125901_9.stderr parsl.dataflow.dflow DEBUG: Task 840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 840 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 840 has memoization hash 4530296900bca00359185b35155ee97a parsl.dataflow.memoization INFO: Task 840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 840 try 0 launched on executor multi with executor id 841 parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/10545506-5212-41e5-b45b-9ccb887877df_isrForCrosstalkSources_1125899_51.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/10545506-5212-41e5-b45b-9ccb887877df_isrForCrosstalkSources_1125899_51.stderr parsl.dataflow.dflow DEBUG: Task 841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 841 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 841 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 841 has memoization hash f91c5dd00bf16b52bdc34830fa8013d8 parsl.dataflow.memoization INFO: Task 841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be389a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 841 try 0 launched on executor multi with executor id 842 parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/7e7d8a45-c860-4e29-8a42-cee3c770d5a8_isrForCrosstalkSources_1125938_30.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/7e7d8a45-c860-4e29-8a42-cee3c770d5a8_isrForCrosstalkSources_1125938_30.stderr parsl.dataflow.dflow DEBUG: Task 842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 842 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 842 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 842 has memoization hash f0e0bdd60e3075bf53530dab28aa085b parsl.dataflow.memoization INFO: Task 842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 842 try 0 launched on executor multi with executor id 843 parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/387174f1-51ab-4ad2-97dc-4bad64fd7072_isrForCrosstalkSources_1125895_7.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/387174f1-51ab-4ad2-97dc-4bad64fd7072_isrForCrosstalkSources_1125895_7.stderr parsl.dataflow.dflow DEBUG: Task 843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 843 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 843 has memoization hash 1d1e09d245f48598e1b0b71d0861fb00 parsl.dataflow.memoization INFO: Task 843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 843 try 0 launched on executor multi with executor id 844 parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/a0a609cf-d137-4ede-810a-236e06f1cff9_isrForCrosstalkSources_1125895_53.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/a0a609cf-d137-4ede-810a-236e06f1cff9_isrForCrosstalkSources_1125895_53.stderr parsl.dataflow.dflow DEBUG: Task 844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 844 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 844 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 844 has memoization hash a7bf9df968831efa31a72af4b7d676e9 parsl.dataflow.memoization INFO: Task 844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 844 try 0 launched on executor multi with executor id 845 parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/dfde5563-5524-484d-b214-12bea7b7e475_isrForCrosstalkSources_1125931_21.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/dfde5563-5524-484d-b214-12bea7b7e475_isrForCrosstalkSources_1125931_21.stderr parsl.dataflow.dflow DEBUG: Task 845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 845 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 845 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 845 has memoization hash e29abe1906353d9256f306a983504e68 parsl.dataflow.memoization INFO: Task 845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 845 try 0 launched on executor multi with executor id 846 parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/aed4bd26-fe85-49a9-b562-b56fe7fa19a8_isrForCrosstalkSources_1125895_56.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/aed4bd26-fe85-49a9-b562-b56fe7fa19a8_isrForCrosstalkSources_1125895_56.stderr parsl.dataflow.dflow DEBUG: Task 846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 846 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 846 has memoization hash 01ca89675a523928d915877cd8f34b94 parsl.dataflow.memoization INFO: Task 846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 846 try 0 launched on executor multi with executor id 847 parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/50bd657c-0c96-4842-a57c-d7771bccc4fb_isrForCrosstalkSources_1125920_11.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/50bd657c-0c96-4842-a57c-d7771bccc4fb_isrForCrosstalkSources_1125920_11.stderr parsl.dataflow.dflow DEBUG: Task 847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 847 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 847 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 847 has memoization hash 178c315b5001e983b57e1b132793ef67 parsl.dataflow.memoization INFO: Task 847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 847 try 0 launched on executor multi with executor id 848 parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/86f5954e-e5b9-4e32-bae7-72fdbf99f56e_isrForCrosstalkSources_1125896_53.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/86f5954e-e5b9-4e32-bae7-72fdbf99f56e_isrForCrosstalkSources_1125896_53.stderr parsl.dataflow.dflow DEBUG: Task 848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 848 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 848 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 848 has memoization hash 4526abdc5b4535da94ad0c17577db187 parsl.dataflow.memoization INFO: Task 848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 848 try 0 launched on executor multi with executor id 849 parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/71fba330-8706-4987-b6d1-5aa97a755acc_isrForCrosstalkSources_1125904_46.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/71fba330-8706-4987-b6d1-5aa97a755acc_isrForCrosstalkSources_1125904_46.stderr parsl.dataflow.dflow DEBUG: Task 849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 849 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 849 has memoization hash 9d7f6ff25ba06aaea8d393bf387e84c3 parsl.dataflow.memoization INFO: Task 849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 849 try 0 launched on executor multi with executor id 850 parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/9aa090db-5ff7-4dd2-a5a9-13c7238dd86c_isrForCrosstalkSources_1125898_6.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/9aa090db-5ff7-4dd2-a5a9-13c7238dd86c_isrForCrosstalkSources_1125898_6.stderr parsl.dataflow.dflow DEBUG: Task 850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 850 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 850 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 850 has memoization hash d5ef7a265f38dd563dba378198a822be parsl.dataflow.memoization INFO: Task 850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 850 try 0 launched on executor multi with executor id 851 parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/cefad942-580e-4548-86cd-9a61a43b5cb1_isrForCrosstalkSources_1125940_59.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/cefad942-580e-4548-86cd-9a61a43b5cb1_isrForCrosstalkSources_1125940_59.stderr parsl.dataflow.dflow DEBUG: Task 851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 851 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 851 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 851 has memoization hash fb76d66c8d65a705958548998469aba7 parsl.dataflow.memoization INFO: Task 851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 851 try 0 launched on executor multi with executor id 852 parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/b612b176-3293-421e-91a5-9d0ed34f5607_isrForCrosstalkSources_1125921_27.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/b612b176-3293-421e-91a5-9d0ed34f5607_isrForCrosstalkSources_1125921_27.stderr parsl.dataflow.dflow DEBUG: Task 852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 852 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 852 has memoization hash d741cd5392e1d6e1f674a86ba467b618 parsl.dataflow.memoization INFO: Task 852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be391c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 852 try 0 launched on executor multi with executor id 853 parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/f0f06a00-ff97-4676-a305-94bfccafb336_isrForCrosstalkSources_1125932_48.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/f0f06a00-ff97-4676-a305-94bfccafb336_isrForCrosstalkSources_1125932_48.stderr parsl.dataflow.dflow DEBUG: Task 853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 853 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 853 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 853 has memoization hash 593f3eff09c073d63c31b043f66d2942 parsl.dataflow.memoization INFO: Task 853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 853 try 0 launched on executor multi with executor id 854 parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/3e1b85cf-188d-4687-a077-ac28430a12dc_isrForCrosstalkSources_1125894_2.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/3e1b85cf-188d-4687-a077-ac28430a12dc_isrForCrosstalkSources_1125894_2.stderr parsl.dataflow.dflow DEBUG: Task 854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 854 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 854 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 854 has memoization hash 57a1cc4ce3b2dee511d3c42ae9c6ef56 parsl.dataflow.memoization INFO: Task 854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be38ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 854 try 0 launched on executor multi with executor id 855 parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/245fd55b-de6d-4382-87c3-ecee7280ab12_isrForCrosstalkSources_1125892_55.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/245fd55b-de6d-4382-87c3-ecee7280ab12_isrForCrosstalkSources_1125892_55.stderr parsl.dataflow.dflow DEBUG: Task 855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 855 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 855 has memoization hash d5b8278284076efe28fbdf2883b09d9e parsl.dataflow.memoization INFO: Task 855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 855 try 0 launched on executor multi with executor id 856 parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/777b2b9b-2032-4dcd-b0c3-8734ac2738a4_isrForCrosstalkSources_1125946_37.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/777b2b9b-2032-4dcd-b0c3-8734ac2738a4_isrForCrosstalkSources_1125946_37.stderr parsl.dataflow.dflow DEBUG: Task 856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 856 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 856 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 856 has memoization hash 29ea9e784c44dd8f86212c29ac22c84b parsl.dataflow.memoization INFO: Task 856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 856 try 0 launched on executor multi with executor id 857 parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/fe6b85ac-07b3-4261-a28d-9b0e201633e5_isrForCrosstalkSources_1125903_61.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/fe6b85ac-07b3-4261-a28d-9b0e201633e5_isrForCrosstalkSources_1125903_61.stderr parsl.dataflow.dflow DEBUG: Task 857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 857 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 857 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 857 has memoization hash 4477707780b3a08afab06565ca88be22 parsl.dataflow.memoization INFO: Task 857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 857 try 0 launched on executor multi with executor id 858 parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/0a3bd108-7f50-4327-acba-84c457e58df4_isrForCrosstalkSources_1125892_36.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/0a3bd108-7f50-4327-acba-84c457e58df4_isrForCrosstalkSources_1125892_36.stderr parsl.dataflow.dflow DEBUG: Task 858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 858 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 858 has memoization hash c4edd7f87719231113514ec68dd1aad2 parsl.dataflow.memoization INFO: Task 858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 858 try 0 launched on executor multi with executor id 859 parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b16d9cfc-f0e6-4fe9-a8f7-2733bfd607ba_isrForCrosstalkSources_1125938_18.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b16d9cfc-f0e6-4fe9-a8f7-2733bfd607ba_isrForCrosstalkSources_1125938_18.stderr parsl.dataflow.dflow DEBUG: Task 859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 859 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 859 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 859 has memoization hash f7aab0e1aff4f95650ad2644b45da836 parsl.dataflow.memoization INFO: Task 859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be393a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 859 try 0 launched on executor multi with executor id 860 parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/794bdd86-ace1-4af2-bfff-95a63e7fcf81_isrForCrosstalkSources_1125926_21.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/794bdd86-ace1-4af2-bfff-95a63e7fcf81_isrForCrosstalkSources_1125926_21.stderr parsl.dataflow.dflow DEBUG: Task 860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 860 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 860 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 860 has memoization hash fff5fade69dc0df6f301df5973d1d1cc parsl.dataflow.memoization INFO: Task 860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 860 try 0 launched on executor multi with executor id 861 parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/9b287236-5f02-4394-9a62-c11bb5f1ca6b_isrForCrosstalkSources_1125933_34.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/9b287236-5f02-4394-9a62-c11bb5f1ca6b_isrForCrosstalkSources_1125933_34.stderr parsl.dataflow.dflow DEBUG: Task 861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 861 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 861 has memoization hash 556a995883f93d01b573be4bf845f550 parsl.dataflow.memoization INFO: Task 861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 861 try 0 launched on executor multi with executor id 862 parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/e2dc0513-2ce7-409a-8674-68123b05e22f_isrForCrosstalkSources_1125945_26.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/e2dc0513-2ce7-409a-8674-68123b05e22f_isrForCrosstalkSources_1125945_26.stderr parsl.dataflow.dflow DEBUG: Task 862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 862 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 862 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 862 has memoization hash 7efdbaafb0bb1ae7df8414aeda7c2f22 parsl.dataflow.memoization INFO: Task 862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 862 try 0 launched on executor multi with executor id 863 parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/aeb293f6-44ca-4884-865b-d667d572a2ed_isrForCrosstalkSources_1125919_61.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/aeb293f6-44ca-4884-865b-d667d572a2ed_isrForCrosstalkSources_1125919_61.stderr parsl.dataflow.dflow DEBUG: Task 863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 863 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 863 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 863 has memoization hash 72b490cd63c2d8461c1d41b6ed0ba470 parsl.dataflow.memoization INFO: Task 863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 863 try 0 launched on executor multi with executor id 864 parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2ea8adda-32e4-465c-b763-20c89ee4f44f_isrForCrosstalkSources_1125898_11.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2ea8adda-32e4-465c-b763-20c89ee4f44f_isrForCrosstalkSources_1125898_11.stderr parsl.dataflow.dflow DEBUG: Task 864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 864 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 864 has memoization hash bc1d795f5b51f17987a2d6e2fa9d5ad4 parsl.dataflow.memoization INFO: Task 864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 864 try 0 launched on executor multi with executor id 865 parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/a6aacb45-4839-420d-83a9-f8f1ceb336ac_isrForCrosstalkSources_1125928_52.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/a6aacb45-4839-420d-83a9-f8f1ceb336ac_isrForCrosstalkSources_1125928_52.stderr parsl.dataflow.dflow DEBUG: Task 865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 865 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 865 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 865 has memoization hash 54b574dca1d501ade3e71b23ce791549 parsl.dataflow.memoization INFO: Task 865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 865 try 0 launched on executor multi with executor id 866 parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/4e35278b-4c4c-490c-b47f-856ecbc25ec9_isrForCrosstalkSources_1125918_54.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/4e35278b-4c4c-490c-b47f-856ecbc25ec9_isrForCrosstalkSources_1125918_54.stderr parsl.dataflow.dflow DEBUG: Task 866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 866 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 866 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 866 has memoization hash 30eb076ebaa4bfba0eb265c3f71ac8ad parsl.dataflow.memoization INFO: Task 866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 866 try 0 launched on executor multi with executor id 867 parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/28567900-20eb-4ca2-a4b4-9ffd2a500225_isrForCrosstalkSources_1125903_12.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/28567900-20eb-4ca2-a4b4-9ffd2a500225_isrForCrosstalkSources_1125903_12.stderr parsl.dataflow.dflow DEBUG: Task 867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 867 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 867 has memoization hash 32b3f8129b7e80b3174f41428098e3ad parsl.dataflow.memoization INFO: Task 867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 867 try 0 launched on executor multi with executor id 868 parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/5bbf1971-ddc4-411d-a3c9-3a541851be42_isrForCrosstalkSources_1125943_27.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/5bbf1971-ddc4-411d-a3c9-3a541851be42_isrForCrosstalkSources_1125943_27.stderr parsl.dataflow.dflow DEBUG: Task 868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 868 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 868 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 868 has memoization hash c1b9ac846468186ba53d793ce124e5f8 parsl.dataflow.memoization INFO: Task 868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 868 try 0 launched on executor multi with executor id 869 parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3c437f67-cb01-4a77-af19-4afb313b937d_isrForCrosstalkSources_1125941_42.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3c437f67-cb01-4a77-af19-4afb313b937d_isrForCrosstalkSources_1125941_42.stderr parsl.dataflow.dflow DEBUG: Task 869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 869 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 869 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 869 has memoization hash d778c313bf29ce8294d50e0602513a18 parsl.dataflow.memoization INFO: Task 869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 869 try 0 launched on executor multi with executor id 870 parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/06f222bb-4413-4dbb-8410-11dae4dbf96e_isrForCrosstalkSources_1125908_17.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/06f222bb-4413-4dbb-8410-11dae4dbf96e_isrForCrosstalkSources_1125908_17.stderr parsl.dataflow.dflow DEBUG: Task 870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 870 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 870 has memoization hash 24cc835540c641bcdfea702eaafda024 parsl.dataflow.memoization INFO: Task 870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 870 try 0 launched on executor multi with executor id 871 parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/e7824198-358e-49a7-a524-0f66c35cb9d5_isrForCrosstalkSources_1125897_61.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/e7824198-358e-49a7-a524-0f66c35cb9d5_isrForCrosstalkSources_1125897_61.stderr parsl.dataflow.dflow DEBUG: Task 871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 871 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 871 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 871 has memoization hash 221333595c22244a00dba5d5688b9fcd parsl.dataflow.memoization INFO: Task 871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 871 try 0 launched on executor multi with executor id 872 parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/54be4924-32e7-47ab-b414-1058786d89db_isrForCrosstalkSources_1125906_31.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/54be4924-32e7-47ab-b414-1058786d89db_isrForCrosstalkSources_1125906_31.stderr parsl.dataflow.dflow DEBUG: Task 872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 872 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 872 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 872 has memoization hash 6c98aad997f910e6912a7e202ef7caab parsl.dataflow.memoization INFO: Task 872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 872 try 0 launched on executor multi with executor id 873 parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/d6e9ca62-e041-4912-a4c2-8f9e798e3e84_isrForCrosstalkSources_1125893_40.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/d6e9ca62-e041-4912-a4c2-8f9e798e3e84_isrForCrosstalkSources_1125893_40.stderr parsl.dataflow.dflow DEBUG: Task 873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 873 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 873 has memoization hash 5fd7dc5bff4289906b379f798f5da746 parsl.dataflow.memoization INFO: Task 873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 873 try 0 launched on executor multi with executor id 874 parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/5af00947-8a83-451c-b94a-be9ec3aa1e56_isrForCrosstalkSources_1125925_24.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/5af00947-8a83-451c-b94a-be9ec3aa1e56_isrForCrosstalkSources_1125925_24.stderr parsl.dataflow.dflow DEBUG: Task 874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 874 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 874 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 874 has memoization hash f34ed689c17041e904f1ccbd360f61cf parsl.dataflow.memoization INFO: Task 874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 874 try 0 launched on executor multi with executor id 875 parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/3b531d86-9b5a-42da-ba58-6bf5255ef070_isrForCrosstalkSources_1125931_38.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/3b531d86-9b5a-42da-ba58-6bf5255ef070_isrForCrosstalkSources_1125931_38.stderr parsl.dataflow.dflow DEBUG: Task 875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 875 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 875 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 875 has memoization hash 23d615dfb45333c89bcea8a81090d1b0 parsl.dataflow.memoization INFO: Task 875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 875 try 0 launched on executor multi with executor id 876 parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d413550f-566d-40de-957e-2d26c23b002b_isrForCrosstalkSources_1125923_27.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d413550f-566d-40de-957e-2d26c23b002b_isrForCrosstalkSources_1125923_27.stderr parsl.dataflow.dflow DEBUG: Task 876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 876 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 876 has memoization hash b609cce042432b15da5d1d938c073818 parsl.dataflow.memoization INFO: Task 876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 876 try 0 launched on executor multi with executor id 877 parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/2eb7eee3-a344-4c10-8a29-e23f31b106c7_isrForCrosstalkSources_1125888_11.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/2eb7eee3-a344-4c10-8a29-e23f31b106c7_isrForCrosstalkSources_1125888_11.stderr parsl.dataflow.dflow DEBUG: Task 877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 877 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 877 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 877 has memoization hash 5d99ddeb6d9ce350b6c065cc8b18dd5d parsl.dataflow.memoization INFO: Task 877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 877 try 0 launched on executor multi with executor id 878 parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/af8480ad-4e23-4586-9258-22784d6ddaf7_isrForCrosstalkSources_1125947_22.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/af8480ad-4e23-4586-9258-22784d6ddaf7_isrForCrosstalkSources_1125947_22.stderr parsl.dataflow.dflow DEBUG: Task 878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 878 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 878 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 878 has memoization hash 8517eb000135ab4b5fa041cde0204590 parsl.dataflow.memoization INFO: Task 878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 878 try 0 launched on executor multi with executor id 879 parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/7eb23229-a3e6-43ea-bdb9-6b8ee71027e5_isrForCrosstalkSources_1125923_45.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/7eb23229-a3e6-43ea-bdb9-6b8ee71027e5_isrForCrosstalkSources_1125923_45.stderr parsl.dataflow.dflow DEBUG: Task 879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 879 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 879 has memoization hash 2c9d9d099e7fef71012e0e190facc928 parsl.dataflow.memoization INFO: Task 879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 879 try 0 launched on executor multi with executor id 880 parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/b8e907cd-e9ec-442f-9191-0c8d346625b3_isrForCrosstalkSources_1125936_25.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/b8e907cd-e9ec-442f-9191-0c8d346625b3_isrForCrosstalkSources_1125936_25.stderr parsl.dataflow.dflow DEBUG: Task 880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 880 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 880 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 880 has memoization hash 29cd2ba174d4bb126c81e103ddc05835 parsl.dataflow.memoization INFO: Task 880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 880 try 0 launched on executor multi with executor id 881 parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/039187db-a8e4-4845-a7fd-f73b45054530_isrForCrosstalkSources_1125933_6.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/039187db-a8e4-4845-a7fd-f73b45054530_isrForCrosstalkSources_1125933_6.stderr parsl.dataflow.dflow DEBUG: Task 881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 881 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 881 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 881 has memoization hash 2a1dbd8d402c96eeed966f59dc21ce96 parsl.dataflow.memoization INFO: Task 881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 881 try 0 launched on executor multi with executor id 882 parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/5366fabf-3af4-43b0-b4c2-b62542cf89c7_isrForCrosstalkSources_1125927_8.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/5366fabf-3af4-43b0-b4c2-b62542cf89c7_isrForCrosstalkSources_1125927_8.stderr parsl.dataflow.dflow DEBUG: Task 882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 882 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 882 has memoization hash f4f52f20b5078e6b9c1743e3e6d09e98 parsl.dataflow.memoization INFO: Task 882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 882 try 0 launched on executor multi with executor id 883 parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/7708a0d7-3f40-4828-99d5-6923b71dc600_isrForCrosstalkSources_1125928_36.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/7708a0d7-3f40-4828-99d5-6923b71dc600_isrForCrosstalkSources_1125928_36.stderr parsl.dataflow.dflow DEBUG: Task 883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 883 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 883 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 883 has memoization hash 95f67cc84e42361702fd058b4e5d831c parsl.dataflow.memoization INFO: Task 883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 883 try 0 launched on executor multi with executor id 884 parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/169d2e0c-eaae-4a73-a51f-27a324df53c4_isrForCrosstalkSources_1125895_43.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/169d2e0c-eaae-4a73-a51f-27a324df53c4_isrForCrosstalkSources_1125895_43.stderr parsl.dataflow.dflow DEBUG: Task 884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 884 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 884 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 884 has memoization hash 8f5e47085069f39e48fd749498137984 parsl.dataflow.memoization INFO: Task 884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 884 try 0 launched on executor multi with executor id 885 parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/c270873f-d7d3-48fa-a2d0-003e3151df64_isrForCrosstalkSources_1125897_62.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/c270873f-d7d3-48fa-a2d0-003e3151df64_isrForCrosstalkSources_1125897_62.stderr parsl.dataflow.dflow DEBUG: Task 885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 885 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 885 has memoization hash 9e3c73dd308f5462b97f9721b310eb9d parsl.dataflow.memoization INFO: Task 885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 885 try 0 launched on executor multi with executor id 886 parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/b134820b-0147-46ed-96ea-21580e36afd3_isrForCrosstalkSources_1125896_12.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/b134820b-0147-46ed-96ea-21580e36afd3_isrForCrosstalkSources_1125896_12.stderr parsl.dataflow.dflow DEBUG: Task 886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 886 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 886 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 886 has memoization hash 2a1cef4b3735de13874660bc21cf1608 parsl.dataflow.memoization INFO: Task 886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 886 try 0 launched on executor multi with executor id 887 parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/b7e0b543-051c-435d-b02f-53f55cbccc53_isrForCrosstalkSources_1125897_27.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/b7e0b543-051c-435d-b02f-53f55cbccc53_isrForCrosstalkSources_1125897_27.stderr parsl.dataflow.dflow DEBUG: Task 887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 887 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 887 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 887 has memoization hash 66b716b0a8d2787b20c867fda005bce5 parsl.dataflow.memoization INFO: Task 887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 887 try 0 launched on executor multi with executor id 888 parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/0c39049a-e2dc-4944-8137-a60f3e465d48_isrForCrosstalkSources_1125895_28.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/0c39049a-e2dc-4944-8137-a60f3e465d48_isrForCrosstalkSources_1125895_28.stderr parsl.dataflow.dflow DEBUG: Task 888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 888 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 888 has memoization hash eff42acce0e163fd44f9f82dab2e7307 parsl.dataflow.memoization INFO: Task 888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 888 try 0 launched on executor multi with executor id 889 parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/823864f8-0bbb-4531-99cf-8dde03b2fdc4_isrForCrosstalkSources_1125907_31.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/823864f8-0bbb-4531-99cf-8dde03b2fdc4_isrForCrosstalkSources_1125907_31.stderr parsl.dataflow.dflow DEBUG: Task 889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 889 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 889 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 889 has memoization hash 8ef56728e5ab21e4ed10a540611e2f6f parsl.dataflow.memoization INFO: Task 889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 889 try 0 launched on executor multi with executor id 890 parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a5460acc-7e93-4f27-b804-ada308068e0e_isrForCrosstalkSources_1125937_49.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a5460acc-7e93-4f27-b804-ada308068e0e_isrForCrosstalkSources_1125937_49.stderr parsl.dataflow.dflow DEBUG: Task 890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 890 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 890 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 890 has memoization hash a7013537015c6565046b8f617d13ae59 parsl.dataflow.memoization INFO: Task 890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 890 try 0 launched on executor multi with executor id 891 parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/22cf15bc-7783-49ce-9204-9908ab6b18fa_isrForCrosstalkSources_1125946_3.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/22cf15bc-7783-49ce-9204-9908ab6b18fa_isrForCrosstalkSources_1125946_3.stderr parsl.dataflow.dflow DEBUG: Task 891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 891 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 891 has memoization hash 3d07c16a9a4d065687c6547c085fc0c1 parsl.dataflow.memoization INFO: Task 891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be394e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 891 try 0 launched on executor multi with executor id 892 parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c5919648-a0b9-443d-b0bf-4c83ac37255a_isrForCrosstalkSources_1125910_13.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c5919648-a0b9-443d-b0bf-4c83ac37255a_isrForCrosstalkSources_1125910_13.stderr parsl.dataflow.dflow DEBUG: Task 892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 892 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 892 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 892 has memoization hash d3410dd5f6e103a59a8e7390efdfc61a parsl.dataflow.memoization INFO: Task 892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be396c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 892 try 0 launched on executor multi with executor id 893 parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/672f59a1-9a80-47cc-90aa-01b0bba548ff_isrForCrosstalkSources_1125929_9.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/672f59a1-9a80-47cc-90aa-01b0bba548ff_isrForCrosstalkSources_1125929_9.stderr parsl.dataflow.dflow DEBUG: Task 893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 893 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 893 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 893 has memoization hash 45717dcaf9942b77e60fc1ec8eb9ab56 parsl.dataflow.memoization INFO: Task 893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 893 try 0 launched on executor multi with executor id 894 parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/77182f57-0547-4042-8ce7-a3430f23e54d_isrForCrosstalkSources_1125897_52.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/77182f57-0547-4042-8ce7-a3430f23e54d_isrForCrosstalkSources_1125897_52.stderr parsl.dataflow.dflow DEBUG: Task 894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 894 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 894 has memoization hash 8a5c72ff3a11ec9528e56249192ea5ac parsl.dataflow.memoization INFO: Task 894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 894 try 0 launched on executor multi with executor id 895 parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/b4643dd2-361b-4b79-8b2c-4d7d61a30b88_isrForCrosstalkSources_1125941_41.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/b4643dd2-361b-4b79-8b2c-4d7d61a30b88_isrForCrosstalkSources_1125941_41.stderr parsl.dataflow.dflow DEBUG: Task 895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 895 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 895 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 895 has memoization hash bc86bffc8e56094481904d5024030525 parsl.dataflow.memoization INFO: Task 895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 895 try 0 launched on executor multi with executor id 896 parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a78e75a0-1013-430f-9bfe-e4d44eaa8edb_isrForCrosstalkSources_1125921_50.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a78e75a0-1013-430f-9bfe-e4d44eaa8edb_isrForCrosstalkSources_1125921_50.stderr parsl.dataflow.dflow DEBUG: Task 896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 896 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 896 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 896 has memoization hash 0ba24469e4a208f73734061c4ce88515 parsl.dataflow.memoization INFO: Task 896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be398a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 896 try 0 launched on executor multi with executor id 897 parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/674fee06-a48d-4cec-92c7-89ffba94c004_isrForCrosstalkSources_1125906_44.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/674fee06-a48d-4cec-92c7-89ffba94c004_isrForCrosstalkSources_1125906_44.stderr parsl.dataflow.dflow DEBUG: Task 897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 897 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 897 has memoization hash d2adf4ceac43ce1923673ca7a043a903 parsl.dataflow.memoization INFO: Task 897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be399e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 897 try 0 launched on executor multi with executor id 898 parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/9423b06e-16f6-4fba-8953-1ec04670e990_isrForCrosstalkSources_1125940_45.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/9423b06e-16f6-4fba-8953-1ec04670e990_isrForCrosstalkSources_1125940_45.stderr parsl.dataflow.dflow DEBUG: Task 898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 898 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 898 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 898 has memoization hash 1d3198de5ddff8f40156ca94dad33eb3 parsl.dataflow.memoization INFO: Task 898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 898 try 0 launched on executor multi with executor id 899 parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/2d40dc3d-a1c6-4176-bd47-a459dc861bc8_isrForCrosstalkSources_1125924_10.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/2d40dc3d-a1c6-4176-bd47-a459dc861bc8_isrForCrosstalkSources_1125924_10.stderr parsl.dataflow.dflow DEBUG: Task 899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 899 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 899 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 899 has memoization hash 5e9807696f70ea2880c5be141369dba3 parsl.dataflow.memoization INFO: Task 899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 899 try 0 launched on executor multi with executor id 900 parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/80ccae37-d16c-4ded-8581-a45fb7209806_isrForCrosstalkSources_1125938_16.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/80ccae37-d16c-4ded-8581-a45fb7209806_isrForCrosstalkSources_1125938_16.stderr parsl.dataflow.dflow DEBUG: Task 900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 900 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 900 has memoization hash c156a72ab2c403e15952b00d32c97800 parsl.dataflow.memoization INFO: Task 900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 900 try 0 launched on executor multi with executor id 901 parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/c98c0fd4-21e8-4862-bd26-102830cd844a_isrForCrosstalkSources_1125937_58.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/c98c0fd4-21e8-4862-bd26-102830cd844a_isrForCrosstalkSources_1125937_58.stderr parsl.dataflow.dflow DEBUG: Task 901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 901 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 901 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 901 has memoization hash b97f7da7f2c2da10494e4650a02c5c24 parsl.dataflow.memoization INFO: Task 901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 901 try 0 launched on executor multi with executor id 902 parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/1e6c2f67-5a13-4dde-b157-4ab5068b5896_isrForCrosstalkSources_1125894_23.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/1e6c2f67-5a13-4dde-b157-4ab5068b5896_isrForCrosstalkSources_1125894_23.stderr parsl.dataflow.dflow DEBUG: Task 902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 902 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 902 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 902 has memoization hash 246d43f6ff5440acb39f247ac4fe5fbb parsl.dataflow.memoization INFO: Task 902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 902 try 0 launched on executor multi with executor id 903 parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3d32d322-4195-4a46-9b06-417bbe7f754e_isrForCrosstalkSources_1125941_14.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3d32d322-4195-4a46-9b06-417bbe7f754e_isrForCrosstalkSources_1125941_14.stderr parsl.dataflow.dflow DEBUG: Task 903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 903 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 903 has memoization hash 64cb5cd0ebbbf0c05bd85600ef9af7ce parsl.dataflow.memoization INFO: Task 903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 903 try 0 launched on executor multi with executor id 904 parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/e89f3755-8671-43d8-9c35-cd2087d42897_isrForCrosstalkSources_1125899_8.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/e89f3755-8671-43d8-9c35-cd2087d42897_isrForCrosstalkSources_1125899_8.stderr parsl.dataflow.dflow DEBUG: Task 904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 904 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 904 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 904 has memoization hash 2c17a13d30294b10fff1c983965cd19d parsl.dataflow.memoization INFO: Task 904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 904 try 0 launched on executor multi with executor id 905 parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/7d10d1d5-fc3c-4bce-a411-879f5180c906_isrForCrosstalkSources_1125897_20.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/7d10d1d5-fc3c-4bce-a411-879f5180c906_isrForCrosstalkSources_1125897_20.stderr parsl.dataflow.dflow DEBUG: Task 905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 905 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 905 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 905 has memoization hash 33875357e31092234e00f660e62877cf parsl.dataflow.memoization INFO: Task 905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 905 try 0 launched on executor multi with executor id 906 parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/d009bf13-66b3-4bbc-bc74-43621da64982_isrForCrosstalkSources_1125933_31.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/d009bf13-66b3-4bbc-bc74-43621da64982_isrForCrosstalkSources_1125933_31.stderr parsl.dataflow.dflow DEBUG: Task 906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 906 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 906 has memoization hash 10d23a04ac2e508ea4b4033e181164cb parsl.dataflow.memoization INFO: Task 906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be39f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 906 try 0 launched on executor multi with executor id 907 parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/c5967a72-9980-4054-9e5a-1710482124c6_isrForCrosstalkSources_1125901_16.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/c5967a72-9980-4054-9e5a-1710482124c6_isrForCrosstalkSources_1125901_16.stderr parsl.dataflow.dflow DEBUG: Task 907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 907 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 907 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 907 has memoization hash bcdf338cf4b0edd3783b2572ff62d079 parsl.dataflow.memoization INFO: Task 907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 907 try 0 launched on executor multi with executor id 908 parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/47a08e0e-78dc-45ed-a56c-3067a6a6f81e_isrForCrosstalkSources_1125892_24.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/47a08e0e-78dc-45ed-a56c-3067a6a6f81e_isrForCrosstalkSources_1125892_24.stderr parsl.dataflow.dflow DEBUG: Task 908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 908 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 908 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 908 has memoization hash 82b8bc1099a854a97ff0a9ad31604f6f parsl.dataflow.memoization INFO: Task 908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 908 try 0 launched on executor multi with executor id 909 parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/000ecf68-f3d7-4270-b044-9d82b114f41a_isrForCrosstalkSources_1125928_47.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/000ecf68-f3d7-4270-b044-9d82b114f41a_isrForCrosstalkSources_1125928_47.stderr parsl.dataflow.dflow DEBUG: Task 909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 909 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 909 has memoization hash 46334d3dc29fed0fb4f0873006c590bb parsl.dataflow.memoization INFO: Task 909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 909 try 0 launched on executor multi with executor id 910 parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/b19597a6-01d7-4adc-b929-36a48f2a5090_isrForCrosstalkSources_1125889_18.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/b19597a6-01d7-4adc-b929-36a48f2a5090_isrForCrosstalkSources_1125889_18.stderr parsl.dataflow.dflow DEBUG: Task 910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 910 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 910 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 910 has memoization hash f30b88fe3f23413dba197a33a5aafc4f parsl.dataflow.memoization INFO: Task 910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 910 try 0 launched on executor multi with executor id 911 parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d8c26114-bfbd-4431-97b3-0e7725af85e2_isrForCrosstalkSources_1125923_61.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d8c26114-bfbd-4431-97b3-0e7725af85e2_isrForCrosstalkSources_1125923_61.stderr parsl.dataflow.dflow DEBUG: Task 911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 911 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 911 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 911 has memoization hash 5fd7ed24b50a4b054fea64d6a594a7f9 parsl.dataflow.memoization INFO: Task 911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 911 try 0 launched on executor multi with executor id 912 parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/ec298cf7-6ac1-4f2e-8a03-4175406ccdf7_isrForCrosstalkSources_1125904_47.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/ec298cf7-6ac1-4f2e-8a03-4175406ccdf7_isrForCrosstalkSources_1125904_47.stderr parsl.dataflow.dflow DEBUG: Task 912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 912 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 912 has memoization hash 304aee88738e3db1fc3623356fa520b9 parsl.dataflow.memoization INFO: Task 912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 912 try 0 launched on executor multi with executor id 913 parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/53ba9cbd-ca01-48ec-b531-3a32d73f202c_isrForCrosstalkSources_1125898_29.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/53ba9cbd-ca01-48ec-b531-3a32d73f202c_isrForCrosstalkSources_1125898_29.stderr parsl.dataflow.dflow DEBUG: Task 913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 913 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 913 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 913 has memoization hash 10e215fc53dc71b297eef943256f985e parsl.dataflow.memoization INFO: Task 913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 913 try 0 launched on executor multi with executor id 914 parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/0bcb33e5-01b4-4b2b-97eb-395e0b494e4c_isrForCrosstalkSources_1125918_21.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/0bcb33e5-01b4-4b2b-97eb-395e0b494e4c_isrForCrosstalkSources_1125918_21.stderr parsl.dataflow.dflow DEBUG: Task 914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 914 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 914 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 914 has memoization hash 14755e04ad8ebdb1c60191567bdbd974 parsl.dataflow.memoization INFO: Task 914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 914 try 0 launched on executor multi with executor id 915 parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/3a3cf5c5-0cc4-4a2a-8572-a5fc807e164b_isrForCrosstalkSources_1125908_39.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/3a3cf5c5-0cc4-4a2a-8572-a5fc807e164b_isrForCrosstalkSources_1125908_39.stderr parsl.dataflow.dflow DEBUG: Task 915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 915 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 915 has memoization hash 84464538c972c613d901981517946cc7 parsl.dataflow.memoization INFO: Task 915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 915 try 0 launched on executor multi with executor id 916 parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/1676fbd7-8bd1-48c3-bdfd-0e306474e18e_isrForCrosstalkSources_1125930_25.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/1676fbd7-8bd1-48c3-bdfd-0e306474e18e_isrForCrosstalkSources_1125930_25.stderr parsl.dataflow.dflow DEBUG: Task 916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 916 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 916 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 916 has memoization hash aaf29e800979e7f3fc721ca203543561 parsl.dataflow.memoization INFO: Task 916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 916 try 0 launched on executor multi with executor id 917 parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/b460cd4a-0aec-4964-9133-6e5a9753fc61_isrForCrosstalkSources_1125940_58.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/b460cd4a-0aec-4964-9133-6e5a9753fc61_isrForCrosstalkSources_1125940_58.stderr parsl.dataflow.dflow DEBUG: Task 917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 917 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 917 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 917 has memoization hash d0392971dcd08859e741e28d9ee7837a parsl.dataflow.memoization INFO: Task 917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 917 try 0 launched on executor multi with executor id 918 parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/4ca0315d-1d2d-446e-9a3c-8b442dca842d_isrForCrosstalkSources_1125889_49.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/4ca0315d-1d2d-446e-9a3c-8b442dca842d_isrForCrosstalkSources_1125889_49.stderr parsl.dataflow.dflow DEBUG: Task 918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 918 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 918 has memoization hash 04a6f8a5f70882acfdbbd6fb6d519b3e parsl.dataflow.memoization INFO: Task 918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 918 try 0 launched on executor multi with executor id 919 parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/93df4af7-b3cb-4860-bf1b-ba064182c48b_isrForCrosstalkSources_1125935_51.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/93df4af7-b3cb-4860-bf1b-ba064182c48b_isrForCrosstalkSources_1125935_51.stderr parsl.dataflow.dflow DEBUG: Task 919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 919 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 919 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 919 has memoization hash a557947786688e0972b5d44f60ed8167 parsl.dataflow.memoization INFO: Task 919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 919 try 0 launched on executor multi with executor id 920 parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/58df12de-83a8-447a-a277-5ddd12fda3ce_isrForCrosstalkSources_1125904_51.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/58df12de-83a8-447a-a277-5ddd12fda3ce_isrForCrosstalkSources_1125904_51.stderr parsl.dataflow.dflow DEBUG: Task 920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 920 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 920 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 920 has memoization hash 85e3bf7636c6a604aa3524d6bb7c483e parsl.dataflow.memoization INFO: Task 920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 920 try 0 launched on executor multi with executor id 921 parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/57092952-f124-41c8-a69c-d4dd3498d960_isrForCrosstalkSources_1125916_62.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/57092952-f124-41c8-a69c-d4dd3498d960_isrForCrosstalkSources_1125916_62.stderr parsl.dataflow.dflow DEBUG: Task 921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 921 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 921 has memoization hash 7235c5e2667aed1a759dbb516aad8179 parsl.dataflow.memoization INFO: Task 921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 921 try 0 launched on executor multi with executor id 922 parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/2a86149f-bee1-49d0-a718-8c86585cd92e_isrForCrosstalkSources_1125892_50.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/2a86149f-bee1-49d0-a718-8c86585cd92e_isrForCrosstalkSources_1125892_50.stderr parsl.dataflow.dflow DEBUG: Task 922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 922 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 922 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 922 has memoization hash 8945b0854b9810ab408dba659fa78e6a parsl.dataflow.memoization INFO: Task 922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 922 try 0 launched on executor multi with executor id 923 parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/8bd58d55-9460-43fa-a66c-a025f66be7c6_isrForCrosstalkSources_1125897_41.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/8bd58d55-9460-43fa-a66c-a025f66be7c6_isrForCrosstalkSources_1125897_41.stderr parsl.dataflow.dflow DEBUG: Task 923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 923 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 923 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 923 has memoization hash f0aab929b45e9f6ce0b661e7c9ea84d1 parsl.dataflow.memoization INFO: Task 923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 923 try 0 launched on executor multi with executor id 924 parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/6c3d9194-e7ec-4f0b-9d93-b456fc846f49_isrForCrosstalkSources_1125933_41.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/6c3d9194-e7ec-4f0b-9d93-b456fc846f49_isrForCrosstalkSources_1125933_41.stderr parsl.dataflow.dflow DEBUG: Task 924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 924 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 924 has memoization hash bdf228bc9c913e29cfeeab3a9bdb96cf parsl.dataflow.memoization INFO: Task 924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 924 try 0 launched on executor multi with executor id 925 parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/8f795c4b-ee5d-44da-8301-74cf599e60c9_isrForCrosstalkSources_1125910_16.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/8f795c4b-ee5d-44da-8301-74cf599e60c9_isrForCrosstalkSources_1125910_16.stderr parsl.dataflow.dflow DEBUG: Task 925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 925 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 925 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 925 has memoization hash 4b3b37b7ad22e0c61da8c388890f2994 parsl.dataflow.memoization INFO: Task 925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3be3a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 925 try 0 launched on executor multi with executor id 926 parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/538b0d68-b701-4813-af9f-ec78fe368713_isrForCrosstalkSources_1125903_42.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/538b0d68-b701-4813-af9f-ec78fe368713_isrForCrosstalkSources_1125903_42.stderr parsl.dataflow.dflow DEBUG: Task 926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 926 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 926 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 926 has memoization hash cdbf7f127771d8a98f3f4f9f0b2809f2 parsl.dataflow.memoization INFO: Task 926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d340e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 926 try 0 launched on executor multi with executor id 927 parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/9ad9a741-ce44-491e-92ec-ea0669533eae_isrForCrosstalkSources_1125931_18.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/9ad9a741-ce44-491e-92ec-ea0669533eae_isrForCrosstalkSources_1125931_18.stderr parsl.dataflow.dflow DEBUG: Task 927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 927 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 927 has memoization hash 94f6f7e77e57ce63959d7c78768381cd parsl.dataflow.memoization INFO: Task 927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 927 try 0 launched on executor multi with executor id 928 parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/5511e4de-58f5-46c4-87c8-e90e75bd3194_isrForCrosstalkSources_1125947_32.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/5511e4de-58f5-46c4-87c8-e90e75bd3194_isrForCrosstalkSources_1125947_32.stderr parsl.dataflow.dflow DEBUG: Task 928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 928 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 928 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 928 has memoization hash 55403c42407510ef57961d31921a690e parsl.dataflow.memoization INFO: Task 928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 928 try 0 launched on executor multi with executor id 929 parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f5ea4ef4-acd7-4adf-9cf4-f918c8cc1094_isrForCrosstalkSources_1125894_35.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f5ea4ef4-acd7-4adf-9cf4-f918c8cc1094_isrForCrosstalkSources_1125894_35.stderr parsl.dataflow.dflow DEBUG: Task 929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 929 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 929 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 929 has memoization hash 52f0183122a0593f0959e0d645b5e424 parsl.dataflow.memoization INFO: Task 929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d342c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 929 try 0 launched on executor multi with executor id 930 parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c70c1e5c-68fe-408d-88f5-307425106330_isrForCrosstalkSources_1125917_28.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c70c1e5c-68fe-408d-88f5-307425106330_isrForCrosstalkSources_1125917_28.stderr parsl.dataflow.dflow DEBUG: Task 930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 930 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 930 has memoization hash 582dccb307be369b83ffdd8b80f69654 parsl.dataflow.memoization INFO: Task 930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 930 try 0 launched on executor multi with executor id 931 parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/8c858ddb-3f30-4d39-a53b-36862e8d3de0_isrForCrosstalkSources_1125895_39.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/8c858ddb-3f30-4d39-a53b-36862e8d3de0_isrForCrosstalkSources_1125895_39.stderr parsl.dataflow.dflow DEBUG: Task 931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 931 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 931 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 931 has memoization hash 7db4a07bf2f1612c1f6c954b95c7fa6f parsl.dataflow.memoization INFO: Task 931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 931 try 0 launched on executor multi with executor id 932 parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/54421c41-f0f3-46ca-9e91-1d603e937167_isrForCrosstalkSources_1125894_32.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/54421c41-f0f3-46ca-9e91-1d603e937167_isrForCrosstalkSources_1125894_32.stderr parsl.dataflow.dflow DEBUG: Task 932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 932 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 932 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 932 has memoization hash c14f641291baa6d9c5c80d9a19328ff8 parsl.dataflow.memoization INFO: Task 932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 932 try 0 launched on executor multi with executor id 933 parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/5e81597f-217c-4dbd-a589-02179455f9d5_isrForCrosstalkSources_1125902_2.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/5e81597f-217c-4dbd-a589-02179455f9d5_isrForCrosstalkSources_1125902_2.stderr parsl.dataflow.dflow DEBUG: Task 933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 933 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 933 has memoization hash cca5fd9e90c9191553585fdda5a8b46c parsl.dataflow.memoization INFO: Task 933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d344a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 933 try 0 launched on executor multi with executor id 934 parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/4186c740-def9-447f-aa03-fc5bcebc03d2_isrForCrosstalkSources_1125911_11.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/4186c740-def9-447f-aa03-fc5bcebc03d2_isrForCrosstalkSources_1125911_11.stderr parsl.dataflow.dflow DEBUG: Task 934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 934 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 934 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 934 has memoization hash 2c0527777000ff8069e7fb7f8d5cddaa parsl.dataflow.memoization INFO: Task 934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d345e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 934 try 0 launched on executor multi with executor id 935 parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/e36b22f3-6d70-43af-960d-bbab4d61ca2c_isrForCrosstalkSources_1125888_45.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/e36b22f3-6d70-43af-960d-bbab4d61ca2c_isrForCrosstalkSources_1125888_45.stderr parsl.dataflow.dflow DEBUG: Task 935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 935 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 935 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 935 has memoization hash c204ac21f03d73a6af17e0b44dec4a25 parsl.dataflow.memoization INFO: Task 935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 935 try 0 launched on executor multi with executor id 936 parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/567a6aa1-5c85-45ea-8a9c-0ed9addcb87b_isrForCrosstalkSources_1125914_9.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/567a6aa1-5c85-45ea-8a9c-0ed9addcb87b_isrForCrosstalkSources_1125914_9.stderr parsl.dataflow.dflow DEBUG: Task 936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 936 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 936 has memoization hash af7f94ecbc711ac62831caa783a099f5 parsl.dataflow.memoization INFO: Task 936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 936 try 0 launched on executor multi with executor id 937 parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/eb5ffebc-2d5f-4312-a59a-c34b50770f15_isrForCrosstalkSources_1125907_62.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/eb5ffebc-2d5f-4312-a59a-c34b50770f15_isrForCrosstalkSources_1125907_62.stderr parsl.dataflow.dflow DEBUG: Task 937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 937 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 937 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 937 has memoization hash b89dba6e6c26f3b619015530bc310c26 parsl.dataflow.memoization INFO: Task 937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 937 try 0 launched on executor multi with executor id 938 parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/4527265d-699d-4ebf-a396-1cdc36f811db_isrForCrosstalkSources_1125897_38.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/4527265d-699d-4ebf-a396-1cdc36f811db_isrForCrosstalkSources_1125897_38.stderr parsl.dataflow.dflow DEBUG: Task 938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 938 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 938 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 938 has memoization hash 87d9631680ca7337251e8270755ad5d0 parsl.dataflow.memoization INFO: Task 938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d347c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 938 try 0 launched on executor multi with executor id 939 parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/396e02df-112c-4a17-a8fb-fa5da178771e_isrForCrosstalkSources_1125941_13.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/396e02df-112c-4a17-a8fb-fa5da178771e_isrForCrosstalkSources_1125941_13.stderr parsl.dataflow.dflow DEBUG: Task 939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 939 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 939 has memoization hash d9fe846df143167826033c3e8d8b4749 parsl.dataflow.memoization INFO: Task 939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 939 try 0 launched on executor multi with executor id 940 parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/ddd87100-d3c0-447a-965c-fa186a7263e5_isrForCrosstalkSources_1125897_11.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/ddd87100-d3c0-447a-965c-fa186a7263e5_isrForCrosstalkSources_1125897_11.stderr parsl.dataflow.dflow DEBUG: Task 940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 940 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 940 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 940 has memoization hash 238aff4f2ea11514720804b87dbe738e parsl.dataflow.memoization INFO: Task 940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d349a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 940 try 0 launched on executor multi with executor id 941 parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/99ad0f22-d1d9-4acf-9237-820473489150_isrForCrosstalkSources_1125908_53.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/99ad0f22-d1d9-4acf-9237-820473489150_isrForCrosstalkSources_1125908_53.stderr parsl.dataflow.dflow DEBUG: Task 941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 941 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 941 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 941 has memoization hash 19fd82c19e5436b853e09448d75939ef parsl.dataflow.memoization INFO: Task 941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 941 try 0 launched on executor multi with executor id 942 parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/6d11d244-79bc-4cec-95b2-d6c097804076_isrForCrosstalkSources_1125944_21.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/6d11d244-79bc-4cec-95b2-d6c097804076_isrForCrosstalkSources_1125944_21.stderr parsl.dataflow.dflow DEBUG: Task 942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 942 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 942 has memoization hash 5df3e85f1b06318c509d9abf2d449b8b parsl.dataflow.memoization INFO: Task 942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 942 try 0 launched on executor multi with executor id 943 parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/81124ccb-14e4-4cf0-9750-7b0d241ca057_isrForCrosstalkSources_1125943_31.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/81124ccb-14e4-4cf0-9750-7b0d241ca057_isrForCrosstalkSources_1125943_31.stderr parsl.dataflow.dflow DEBUG: Task 943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 943 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 943 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 943 has memoization hash 3d45d117ab4e761baa4210626a662c28 parsl.dataflow.memoization INFO: Task 943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 943 try 0 launched on executor multi with executor id 944 parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/0db647db-9e8e-4f2d-b2d0-01bb5a3bae8f_isrForCrosstalkSources_1125925_12.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/0db647db-9e8e-4f2d-b2d0-01bb5a3bae8f_isrForCrosstalkSources_1125925_12.stderr parsl.dataflow.dflow DEBUG: Task 944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 944 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 944 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 944 has memoization hash 10fb0853804e1bbbf003006d4cc4c36f parsl.dataflow.memoization INFO: Task 944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 944 try 0 launched on executor multi with executor id 945 parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/59994020-8f18-42ad-8ada-507b2c22327f_isrForCrosstalkSources_1125946_8.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/59994020-8f18-42ad-8ada-507b2c22327f_isrForCrosstalkSources_1125946_8.stderr parsl.dataflow.dflow DEBUG: Task 945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 945 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 945 has memoization hash 3d2c0999bfe5cfc117c72c78a49f21d8 parsl.dataflow.memoization INFO: Task 945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 945 try 0 launched on executor multi with executor id 946 parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/c166e308-38c0-42fe-94c1-a9d7ac6cafdd_isrForCrosstalkSources_1125909_38.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/c166e308-38c0-42fe-94c1-a9d7ac6cafdd_isrForCrosstalkSources_1125909_38.stderr parsl.dataflow.dflow DEBUG: Task 946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 946 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 946 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 946 has memoization hash 088579552df83940b2abced3300b98a8 parsl.dataflow.memoization INFO: Task 946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 946 try 0 launched on executor multi with executor id 947 parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0d87e29d-78dd-40cf-bf8e-561f4c7bc183_isrForCrosstalkSources_1125890_13.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0d87e29d-78dd-40cf-bf8e-561f4c7bc183_isrForCrosstalkSources_1125890_13.stderr parsl.dataflow.dflow DEBUG: Task 947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 947 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 947 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 947 has memoization hash c498fbcdfcfdfdc0f173422e2a409835 parsl.dataflow.memoization INFO: Task 947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 947 try 0 launched on executor multi with executor id 948 parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8bcfb6b2-e09b-45c3-830c-40ac2ce6d9c9_isrForCrosstalkSources_1125944_50.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8bcfb6b2-e09b-45c3-830c-40ac2ce6d9c9_isrForCrosstalkSources_1125944_50.stderr parsl.dataflow.dflow DEBUG: Task 948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 948 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 948 has memoization hash 666939dd2cd22a38d65a19fb08c19530 parsl.dataflow.memoization INFO: Task 948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 948 try 0 launched on executor multi with executor id 949 parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/620774ff-7652-4d51-bc88-5a5dfb15bd6d_isrForCrosstalkSources_1125946_57.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/620774ff-7652-4d51-bc88-5a5dfb15bd6d_isrForCrosstalkSources_1125946_57.stderr parsl.dataflow.dflow DEBUG: Task 949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 949 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 949 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 949 has memoization hash 048fda621c95d602653756e9ab807283 parsl.dataflow.memoization INFO: Task 949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 949 try 0 launched on executor multi with executor id 950 parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/7f006787-353c-43e4-b220-dd3a7b9008ce_isrForCrosstalkSources_1125900_41.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/7f006787-353c-43e4-b220-dd3a7b9008ce_isrForCrosstalkSources_1125900_41.stderr parsl.dataflow.dflow DEBUG: Task 950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 950 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 950 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 950 has memoization hash 6884a5ec0240ec13a47630383b326834 parsl.dataflow.memoization INFO: Task 950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 950 try 0 launched on executor multi with executor id 951 parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/3454be08-e5de-4c3c-ac23-24fba7b98625_isrForCrosstalkSources_1125945_25.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/3454be08-e5de-4c3c-ac23-24fba7b98625_isrForCrosstalkSources_1125945_25.stderr parsl.dataflow.dflow DEBUG: Task 951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 951 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 951 has memoization hash 27f26597af115a37dc4b966c15a24e7a parsl.dataflow.memoization INFO: Task 951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 951 try 0 launched on executor multi with executor id 952 parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/f9571479-3379-4fc5-aef9-98a0c9d6e9ac_isrForCrosstalkSources_1125890_32.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/f9571479-3379-4fc5-aef9-98a0c9d6e9ac_isrForCrosstalkSources_1125890_32.stderr parsl.dataflow.dflow DEBUG: Task 952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 952 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 952 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 952 has memoization hash 8d1ca88ad5295bd22df44c48f3ed2e23 parsl.dataflow.memoization INFO: Task 952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d351c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 952 try 0 launched on executor multi with executor id 953 parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/a5244f68-8152-44a7-8685-36d7013c22b3_isrForCrosstalkSources_1125910_23.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/a5244f68-8152-44a7-8685-36d7013c22b3_isrForCrosstalkSources_1125910_23.stderr parsl.dataflow.dflow DEBUG: Task 953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 953 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 953 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 953 has memoization hash 00ac38038ccf4d5f7f1a553a54957e97 parsl.dataflow.memoization INFO: Task 953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 953 try 0 launched on executor multi with executor id 954 parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/9709dcb6-2f49-4f1e-9f17-c0155f104f00_isrForCrosstalkSources_1125927_3.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/9709dcb6-2f49-4f1e-9f17-c0155f104f00_isrForCrosstalkSources_1125927_3.stderr parsl.dataflow.dflow DEBUG: Task 954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 954 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 954 has memoization hash 38197dd447d66dfbeef2f3de097b0583 parsl.dataflow.memoization INFO: Task 954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 954 try 0 launched on executor multi with executor id 955 parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/4743aa7f-31e0-46f4-8169-4df6b500f067_isrForCrosstalkSources_1125945_14.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/4743aa7f-31e0-46f4-8169-4df6b500f067_isrForCrosstalkSources_1125945_14.stderr parsl.dataflow.dflow DEBUG: Task 955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 955 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 955 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 955 has memoization hash 59d602aadd18ecc956499bacebffedb7 parsl.dataflow.memoization INFO: Task 955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 955 try 0 launched on executor multi with executor id 956 parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/3c5c1a06-0892-422c-b0ca-e5307ff3f6a8_isrForCrosstalkSources_1125910_4.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/3c5c1a06-0892-422c-b0ca-e5307ff3f6a8_isrForCrosstalkSources_1125910_4.stderr parsl.dataflow.dflow DEBUG: Task 956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 956 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 956 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 956 has memoization hash 154e6f7ff17c7ee6a35d3d2735f85df1 parsl.dataflow.memoization INFO: Task 956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d353a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 956 try 0 launched on executor multi with executor id 957 parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/492665a7-0c30-4f63-b292-7d07f8536cf2_isrForCrosstalkSources_1125915_16.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/492665a7-0c30-4f63-b292-7d07f8536cf2_isrForCrosstalkSources_1125915_16.stderr parsl.dataflow.dflow DEBUG: Task 957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 957 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 957 has memoization hash 572328a3104cffeddd658de71b6e7224 parsl.dataflow.memoization INFO: Task 957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d354e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 957 try 0 launched on executor multi with executor id 958 parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/66b05aee-8953-4e03-8be9-d3400cbc97ec_isrForCrosstalkSources_1125888_16.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/66b05aee-8953-4e03-8be9-d3400cbc97ec_isrForCrosstalkSources_1125888_16.stderr parsl.dataflow.dflow DEBUG: Task 958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 958 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 958 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 958 has memoization hash 697cfccd430445f26370b25fda07d2e1 parsl.dataflow.memoization INFO: Task 958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 958 try 0 launched on executor multi with executor id 959 parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/2c791773-fd38-44b8-b071-f047679bc253_isrForCrosstalkSources_1125904_50.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/2c791773-fd38-44b8-b071-f047679bc253_isrForCrosstalkSources_1125904_50.stderr parsl.dataflow.dflow DEBUG: Task 959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 959 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 959 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 959 has memoization hash e42d9f032dbb141ef3098f54255f59d5 parsl.dataflow.memoization INFO: Task 959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 959 try 0 launched on executor multi with executor id 960 parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3c46968a-6985-4854-ac91-42d53d67a573_isrForCrosstalkSources_1125912_60.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3c46968a-6985-4854-ac91-42d53d67a573_isrForCrosstalkSources_1125912_60.stderr parsl.dataflow.dflow DEBUG: Task 960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 960 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 960 has memoization hash 16df4b190dee9a378d3c4d8937101c51 parsl.dataflow.memoization INFO: Task 960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 960 try 0 launched on executor multi with executor id 961 parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/3dfbf906-85e7-4a87-894f-45cdb6e653de_isrForCrosstalkSources_1125922_50.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/3dfbf906-85e7-4a87-894f-45cdb6e653de_isrForCrosstalkSources_1125922_50.stderr parsl.dataflow.dflow DEBUG: Task 961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 961 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 961 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 961 has memoization hash 468f0e511307f77f3e260aed03af193f parsl.dataflow.memoization INFO: Task 961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d356c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 961 try 0 launched on executor multi with executor id 962 parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/7d0e4d18-262d-4e2f-af01-73e933c8568f_isrForCrosstalkSources_1125894_51.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/7d0e4d18-262d-4e2f-af01-73e933c8568f_isrForCrosstalkSources_1125894_51.stderr parsl.dataflow.dflow DEBUG: Task 962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 962 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 962 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 962 has memoization hash 5f8333c23dca5c39bc5b1c2d6c7f19ed parsl.dataflow.memoization INFO: Task 962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 962 try 0 launched on executor multi with executor id 963 parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/d6e615fe-477c-408a-8ef6-c458e3a903a2_isrForCrosstalkSources_1125922_7.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/d6e615fe-477c-408a-8ef6-c458e3a903a2_isrForCrosstalkSources_1125922_7.stderr parsl.dataflow.dflow DEBUG: Task 963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 963 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 963 has memoization hash 1df88ed6d2212eec0e7b52cd384316e2 parsl.dataflow.memoization INFO: Task 963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 963 try 0 launched on executor multi with executor id 964 parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/5e9ba6f6-3cb7-4699-ae2c-9e6be647934d_isrForCrosstalkSources_1125944_29.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/5e9ba6f6-3cb7-4699-ae2c-9e6be647934d_isrForCrosstalkSources_1125944_29.stderr parsl.dataflow.dflow DEBUG: Task 964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 964 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 964 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 964 has memoization hash 02a5b9ca49cd7feb0ab5ffa5bd1ad37b parsl.dataflow.memoization INFO: Task 964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d358a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 964 try 0 launched on executor multi with executor id 965 parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/97ef57b5-d20f-4798-aaca-d2da90220b49_isrForCrosstalkSources_1125921_51.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/97ef57b5-d20f-4798-aaca-d2da90220b49_isrForCrosstalkSources_1125921_51.stderr parsl.dataflow.dflow DEBUG: Task 965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 965 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 965 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 965 has memoization hash 2dfb30a8e6504c3d329f78eb3bcbdd56 parsl.dataflow.memoization INFO: Task 965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 965 try 0 launched on executor multi with executor id 966 parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/ef1c2832-0a41-40ba-8720-20e59b664227_isrForCrosstalkSources_1125905_41.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/ef1c2832-0a41-40ba-8720-20e59b664227_isrForCrosstalkSources_1125905_41.stderr parsl.dataflow.dflow DEBUG: Task 966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 966 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 966 has memoization hash 53861af8556197862f0c3b2d8af31cc8 parsl.dataflow.memoization INFO: Task 966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d359e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 966 try 0 launched on executor multi with executor id 967 parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/ad9ded94-d000-47b0-afe1-8716d9c9ea89_isrForCrosstalkSources_1125941_31.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/ad9ded94-d000-47b0-afe1-8716d9c9ea89_isrForCrosstalkSources_1125941_31.stderr parsl.dataflow.dflow DEBUG: Task 967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 967 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 967 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 967 has memoization hash 72c1ba2467ec23ec4c976cb37bcaac17 parsl.dataflow.memoization INFO: Task 967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 967 try 0 launched on executor multi with executor id 968 parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/f501ff70-c9da-4c8c-b490-10412234c39b_isrForCrosstalkSources_1125900_28.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/f501ff70-c9da-4c8c-b490-10412234c39b_isrForCrosstalkSources_1125900_28.stderr parsl.dataflow.dflow DEBUG: Task 968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 968 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 968 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 968 has memoization hash a72de7ae505ed7625cc18948185c0775 parsl.dataflow.memoization INFO: Task 968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 968 try 0 launched on executor multi with executor id 969 parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/d757fab5-3dbd-4614-90bd-0cd6657df766_isrForCrosstalkSources_1125896_4.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/d757fab5-3dbd-4614-90bd-0cd6657df766_isrForCrosstalkSources_1125896_4.stderr parsl.dataflow.dflow DEBUG: Task 969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 969 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 969 has memoization hash 5385c93cb277fa3f27f923c381791adf parsl.dataflow.memoization INFO: Task 969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 969 try 0 launched on executor multi with executor id 970 parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/f6493a5a-6826-43f4-af98-187dfb13c85e_isrForCrosstalkSources_1125924_17.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/f6493a5a-6826-43f4-af98-187dfb13c85e_isrForCrosstalkSources_1125924_17.stderr parsl.dataflow.dflow DEBUG: Task 970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 970 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 970 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 970 has memoization hash 58f335382572526c47112318033265ee parsl.dataflow.memoization INFO: Task 970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 970 try 0 launched on executor multi with executor id 971 parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/a7d4c9e3-a645-457c-92a0-0cca47049c34_isrForCrosstalkSources_1125927_47.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/a7d4c9e3-a645-457c-92a0-0cca47049c34_isrForCrosstalkSources_1125927_47.stderr parsl.dataflow.dflow DEBUG: Task 971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 971 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 971 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 971 has memoization hash bb182696430fc637ac20e7a7bd79a854 parsl.dataflow.memoization INFO: Task 971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 971 try 0 launched on executor multi with executor id 972 parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/e841e1e0-d687-4956-a63e-f1aeb92dbd57_isrForCrosstalkSources_1125890_55.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/e841e1e0-d687-4956-a63e-f1aeb92dbd57_isrForCrosstalkSources_1125890_55.stderr parsl.dataflow.dflow DEBUG: Task 972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 972 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 972 has memoization hash 22c2e134b33b5c0c863fe80a0c11d7ff parsl.dataflow.memoization INFO: Task 972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 972 try 0 launched on executor multi with executor id 973 parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/1c2d13d7-2a8d-4382-b72d-10271f756c2b_isrForCrosstalkSources_1125943_20.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/1c2d13d7-2a8d-4382-b72d-10271f756c2b_isrForCrosstalkSources_1125943_20.stderr parsl.dataflow.dflow DEBUG: Task 973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 973 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 973 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 973 has memoization hash 8eac6534ed26b4aa605f36c93aada078 parsl.dataflow.memoization INFO: Task 973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 973 try 0 launched on executor multi with executor id 974 parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/2b64fdc1-f03b-4cc6-acf7-e38ea0722208_isrForCrosstalkSources_1125888_30.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/2b64fdc1-f03b-4cc6-acf7-e38ea0722208_isrForCrosstalkSources_1125888_30.stderr parsl.dataflow.dflow DEBUG: Task 974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 974 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 974 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 974 has memoization hash dbd068bdcafcd3e971f351164f1a9db7 parsl.dataflow.memoization INFO: Task 974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 974 try 0 launched on executor multi with executor id 975 parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/4f2a4e81-3e03-400e-80e4-757c56faa7c0_isrForCrosstalkSources_1125913_62.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/4f2a4e81-3e03-400e-80e4-757c56faa7c0_isrForCrosstalkSources_1125913_62.stderr parsl.dataflow.dflow DEBUG: Task 975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 975 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 975 has memoization hash 0b4beef6ea9a18fc7c0631ff82069851 parsl.dataflow.memoization INFO: Task 975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d35f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 975 try 0 launched on executor multi with executor id 976 parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/16dea266-3a0f-486e-a50c-28cbee3c44d1_isrForCrosstalkSources_1125903_14.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/16dea266-3a0f-486e-a50c-28cbee3c44d1_isrForCrosstalkSources_1125903_14.stderr parsl.dataflow.dflow DEBUG: Task 976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 976 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 976 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 976 has memoization hash 607431e81ef65a0cc2b2c7457e3b0515 parsl.dataflow.memoization INFO: Task 976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 976 try 0 launched on executor multi with executor id 977 parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/a98c7f0a-4163-4e63-aa56-77518e9e350c_isrForCrosstalkSources_1125931_61.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/a98c7f0a-4163-4e63-aa56-77518e9e350c_isrForCrosstalkSources_1125931_61.stderr parsl.dataflow.dflow DEBUG: Task 977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 977 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 977 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 977 has memoization hash 8bebfd4a38a2380775eb711e44e82138 parsl.dataflow.memoization INFO: Task 977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 977 try 0 launched on executor multi with executor id 978 parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/646010e5-a057-4bb2-8319-b711fede2aa7_isrForCrosstalkSources_1125888_10.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/646010e5-a057-4bb2-8319-b711fede2aa7_isrForCrosstalkSources_1125888_10.stderr parsl.dataflow.dflow DEBUG: Task 978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 978 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 978 has memoization hash a15e8fa4159ac0589e9c740944c01a91 parsl.dataflow.memoization INFO: Task 978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d360c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 978 try 0 launched on executor multi with executor id 979 parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/94adf543-569d-423c-8541-6464c00bb64c_isrForCrosstalkSources_1125906_7.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/94adf543-569d-423c-8541-6464c00bb64c_isrForCrosstalkSources_1125906_7.stderr parsl.dataflow.dflow DEBUG: Task 979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 979 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 979 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 979 has memoization hash 8e02013b7e56e32666fc10e5a4801668 parsl.dataflow.memoization INFO: Task 979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 979 try 0 launched on executor multi with executor id 980 parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/b36aadd1-a013-422f-8440-e584603496b3_isrForCrosstalkSources_1125923_29.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/b36aadd1-a013-422f-8440-e584603496b3_isrForCrosstalkSources_1125923_29.stderr parsl.dataflow.dflow DEBUG: Task 980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 980 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 980 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 980 has memoization hash 8f5adfc577683dbe06eb25324c339358 parsl.dataflow.memoization INFO: Task 980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d362a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 980 try 0 launched on executor multi with executor id 981 parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d8d9af40-e133-43f3-997d-455cc0a6c88a_isrForCrosstalkSources_1125923_31.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d8d9af40-e133-43f3-997d-455cc0a6c88a_isrForCrosstalkSources_1125923_31.stderr parsl.dataflow.dflow DEBUG: Task 981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 981 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 981 has memoization hash 4f6b43999f3fba480fbe09bcd148bcae parsl.dataflow.memoization INFO: Task 981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 981 try 0 launched on executor multi with executor id 982 parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b9b84359-8784-402c-bc24-a0a3606b46cb_isrForCrosstalkSources_1125934_25.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b9b84359-8784-402c-bc24-a0a3606b46cb_isrForCrosstalkSources_1125934_25.stderr parsl.dataflow.dflow DEBUG: Task 982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 982 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 982 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 982 has memoization hash 40f4b62382e5ff877a00ccd2909588fd parsl.dataflow.memoization INFO: Task 982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 982 try 0 launched on executor multi with executor id 983 parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/8dd6ab10-965f-44fc-aa70-97f5d3f6c05b_isrForCrosstalkSources_1125912_1.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/8dd6ab10-965f-44fc-aa70-97f5d3f6c05b_isrForCrosstalkSources_1125912_1.stderr parsl.dataflow.dflow DEBUG: Task 983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 983 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 983 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 983 has memoization hash e8801fb29d4643d34b46eb53ebc3ad13 parsl.dataflow.memoization INFO: Task 983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d363e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 983 try 0 launched on executor multi with executor id 984 parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/27bd975c-f176-4652-81b2-86bbdf7dd5c7_isrForCrosstalkSources_1125933_3.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/27bd975c-f176-4652-81b2-86bbdf7dd5c7_isrForCrosstalkSources_1125933_3.stderr parsl.dataflow.dflow DEBUG: Task 984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 984 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 984 has memoization hash cbb4efc2086b320580223753c139ec5c parsl.dataflow.memoization INFO: Task 984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 984 try 0 launched on executor multi with executor id 985 parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/a2322d84-e00b-4a40-be86-c661ea3bf636_isrForCrosstalkSources_1125893_37.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/a2322d84-e00b-4a40-be86-c661ea3bf636_isrForCrosstalkSources_1125893_37.stderr parsl.dataflow.dflow DEBUG: Task 985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 985 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 985 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 985 has memoization hash 89ed3fdfabd32fbdad7f30d31a871aa6 parsl.dataflow.memoization INFO: Task 985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d365c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 985 try 0 launched on executor multi with executor id 986 parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/ae49f39e-b649-44f8-a0b0-cded87704076_isrForCrosstalkSources_1125899_17.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/ae49f39e-b649-44f8-a0b0-cded87704076_isrForCrosstalkSources_1125899_17.stderr parsl.dataflow.dflow DEBUG: Task 986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 986 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 986 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 986 has memoization hash c902eabe98018dbb66db7b877b15d03e parsl.dataflow.memoization INFO: Task 986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 986 try 0 launched on executor multi with executor id 987 parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/67dbb3cb-8cb2-4417-b551-6a17a2bc0791_isrForCrosstalkSources_1125925_29.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/67dbb3cb-8cb2-4417-b551-6a17a2bc0791_isrForCrosstalkSources_1125925_29.stderr parsl.dataflow.dflow DEBUG: Task 987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 987 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 987 has memoization hash c6acccfeedd4e1c1bf9ec31ecbb0998d parsl.dataflow.memoization INFO: Task 987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d367a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 987 try 0 launched on executor multi with executor id 988 parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/409a2de7-ae10-4acf-a795-1f45e98ef1e2_isrForCrosstalkSources_1125901_45.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/409a2de7-ae10-4acf-a795-1f45e98ef1e2_isrForCrosstalkSources_1125901_45.stderr parsl.dataflow.dflow DEBUG: Task 988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 988 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 988 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 988 has memoization hash 22fa3eb99d0e029335b8b88c0235482e parsl.dataflow.memoization INFO: Task 988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 988 try 0 launched on executor multi with executor id 989 parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/549df12a-a449-4e2d-9c04-5b1a97e956ee_isrForCrosstalkSources_1125940_60.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/549df12a-a449-4e2d-9c04-5b1a97e956ee_isrForCrosstalkSources_1125940_60.stderr parsl.dataflow.dflow DEBUG: Task 989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 989 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 989 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 989 has memoization hash cfb4a244bbb2c5cf0e84ecd55f776451 parsl.dataflow.memoization INFO: Task 989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 989 try 0 launched on executor multi with executor id 990 parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/be2bb421-008c-4ca7-8acc-5601b12b4305_isrForCrosstalkSources_1125897_37.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/be2bb421-008c-4ca7-8acc-5601b12b4305_isrForCrosstalkSources_1125897_37.stderr parsl.dataflow.dflow DEBUG: Task 990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 990 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 990 has memoization hash 9a20d4dc3578641b0813e5fbe81203d1 parsl.dataflow.memoization INFO: Task 990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d368e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 990 try 0 launched on executor multi with executor id 991 parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/0b4a19e4-ee42-4cbf-9a92-c06497fb8274_isrForCrosstalkSources_1125894_30.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/0b4a19e4-ee42-4cbf-9a92-c06497fb8274_isrForCrosstalkSources_1125894_30.stderr parsl.dataflow.dflow DEBUG: Task 991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 991 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 991 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 991 has memoization hash 428d9df228a04d223f9802c9833aa164 parsl.dataflow.memoization INFO: Task 991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:44 proc_lsst.multi:170 [INFO] [multi] job_id 16896395 proc_lsst.multi INFO: [multi] job_id 16896395 2024-03-07 15:06:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 15:06:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896395 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896395 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16896395 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-07 15:06:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-07 15:06:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 15:06:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 991 try 0 launched on executor multi with executor id 992 parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/42ed3622-fa9d-4ecd-9b90-c5c8580bd1fa_isrForCrosstalkSources_1125937_23.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/42ed3622-fa9d-4ecd-9b90-c5c8580bd1fa_isrForCrosstalkSources_1125937_23.stderr parsl.dataflow.dflow DEBUG: Task 992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 992 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 992 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 992 has memoization hash cff76aca7428c5ce4ec0557f0b8132d6 parsl.dataflow.memoization INFO: Task 992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 992 try 0 launched on executor multi with executor id 993 parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/8e1e64d8-eb46-4368-9cb7-16a663c80d10_isrForCrosstalkSources_1125925_30.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/8e1e64d8-eb46-4368-9cb7-16a663c80d10_isrForCrosstalkSources_1125925_30.stderr parsl.dataflow.dflow DEBUG: Task 993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 993 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 993 has memoization hash 99e1e8914bf4a049f66d55dfe494143f parsl.dataflow.memoization INFO: Task 993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 993 try 0 launched on executor multi with executor id 994 parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/77b17ec1-6dba-4403-8a5b-e6e674dce83f_isrForCrosstalkSources_1125889_15.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/77b17ec1-6dba-4403-8a5b-e6e674dce83f_isrForCrosstalkSources_1125889_15.stderr parsl.dataflow.dflow DEBUG: Task 994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 994 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 994 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 994 has memoization hash 4918accc2769922e2be17b9aa77d222b parsl.dataflow.memoization INFO: Task 994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 994 try 0 launched on executor multi with executor id 995 parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6b17e937-22ca-490d-91e0-2c31a6f3dff2_isrForCrosstalkSources_1125909_29.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6b17e937-22ca-490d-91e0-2c31a6f3dff2_isrForCrosstalkSources_1125909_29.stderr parsl.dataflow.dflow DEBUG: Task 995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 995 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 995 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 995 has memoization hash ed6a0726852ed3205a3710ce7af3136b parsl.dataflow.memoization INFO: Task 995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 995 try 0 launched on executor multi with executor id 996 parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/fc1769ef-5f43-43d3-8003-e5abb37b29a0_isrForCrosstalkSources_1125938_13.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/fc1769ef-5f43-43d3-8003-e5abb37b29a0_isrForCrosstalkSources_1125938_13.stderr parsl.dataflow.dflow DEBUG: Task 996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 996 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 996 has memoization hash 6a7f692cd0b57fa535a9b08c23a0f28b parsl.dataflow.memoization INFO: Task 996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d34040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 996 try 0 launched on executor multi with executor id 997 parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/ef608198-bd2e-4a69-ac96-38f3c34874f3_isrForCrosstalkSources_1125926_12.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/ef608198-bd2e-4a69-ac96-38f3c34874f3_isrForCrosstalkSources_1125926_12.stderr parsl.dataflow.dflow DEBUG: Task 997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 997 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 997 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 997 has memoization hash 6779b3c3c3a3b22f7a990a91e0057af2 parsl.dataflow.memoization INFO: Task 997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 997 try 0 launched on executor multi with executor id 998 parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/aeda80f8-b25d-4752-864c-3102c2ffadbc_isrForCrosstalkSources_1125930_46.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/aeda80f8-b25d-4752-864c-3102c2ffadbc_isrForCrosstalkSources_1125930_46.stderr parsl.dataflow.dflow DEBUG: Task 998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 998 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 998 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 998 has memoization hash e034d70f44911521483c65ee28e6066c parsl.dataflow.memoization INFO: Task 998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 998 try 0 launched on executor multi with executor id 999 parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/fa5e7ae7-84c1-4a66-ab60-88feb57b550b_isrForCrosstalkSources_1125903_23.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/fa5e7ae7-84c1-4a66-ab60-88feb57b550b_isrForCrosstalkSources_1125903_23.stderr parsl.dataflow.dflow DEBUG: Task 999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 999 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 999 has memoization hash f75a9217d3a56f1325123ac5eb840edb parsl.dataflow.memoization INFO: Task 999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 999 try 0 launched on executor multi with executor id 1000 parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/2640e34b-6276-4029-8372-8fa40a2fe72a_isrForCrosstalkSources_1125927_39.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/2640e34b-6276-4029-8372-8fa40a2fe72a_isrForCrosstalkSources_1125927_39.stderr parsl.dataflow.dflow DEBUG: Task 1000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1000 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1000 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1000 has memoization hash a615ac96a0ace5185cdc26b66f32a2eb parsl.dataflow.memoization INFO: Task 1000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1000 try 0 launched on executor multi with executor id 1001 parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/d86f9177-eb1f-4400-bda7-91e2280d6841_isrForCrosstalkSources_1125894_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/d86f9177-eb1f-4400-bda7-91e2280d6841_isrForCrosstalkSources_1125894_55.stderr parsl.dataflow.dflow DEBUG: Task 1001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1001 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1001 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1001 has memoization hash 4e738ce1d2dae73fe0bc14bc09dd54e5 parsl.dataflow.memoization INFO: Task 1001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d371a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1001 try 0 launched on executor multi with executor id 1002 parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/1f9fa766-e70c-4e79-a7a3-644e005194ad_isrForCrosstalkSources_1125903_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/1f9fa766-e70c-4e79-a7a3-644e005194ad_isrForCrosstalkSources_1125903_43.stderr parsl.dataflow.dflow DEBUG: Task 1002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1002 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1002 has memoization hash ac861bfa495d6b1accaa393a6f4df31b parsl.dataflow.memoization INFO: Task 1002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1002 try 0 launched on executor multi with executor id 1003 parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3e42d99a-bc76-48b2-ae3c-29a72e4ba2de_isrForCrosstalkSources_1125941_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3e42d99a-bc76-48b2-ae3c-29a72e4ba2de_isrForCrosstalkSources_1125941_15.stderr parsl.dataflow.dflow DEBUG: Task 1003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1003 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1003 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1003 has memoization hash fbad35d0cb5796e7f007d5317d31292c parsl.dataflow.memoization INFO: Task 1003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d372e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1003 try 0 launched on executor multi with executor id 1004 parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0c84a530-77ca-45d9-b689-1456b9bfd9b5_isrForCrosstalkSources_1125904_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0c84a530-77ca-45d9-b689-1456b9bfd9b5_isrForCrosstalkSources_1125904_22.stderr parsl.dataflow.dflow DEBUG: Task 1004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1004 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1004 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1004 has memoization hash 526f97221215ac5dce90916458beb7fb parsl.dataflow.memoization INFO: Task 1004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1004 try 0 launched on executor multi with executor id 1005 parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/7daef095-6e44-49f5-9ec8-6c759c6cb8ba_isrForCrosstalkSources_1125925_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/7daef095-6e44-49f5-9ec8-6c759c6cb8ba_isrForCrosstalkSources_1125925_8.stderr parsl.dataflow.dflow DEBUG: Task 1005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1005 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1005 has memoization hash 06d6e94b11468106f5e40c8a32f3ca79 parsl.dataflow.memoization INFO: Task 1005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1005 try 0 launched on executor multi with executor id 1006 parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/e7415e7f-ce47-44cc-8779-26a28d8801ef_isrForCrosstalkSources_1125914_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/e7415e7f-ce47-44cc-8779-26a28d8801ef_isrForCrosstalkSources_1125914_39.stderr parsl.dataflow.dflow DEBUG: Task 1006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1006 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1006 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1006 has memoization hash e60be2b288c64a446a9aa3f61478fa05 parsl.dataflow.memoization INFO: Task 1006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d374c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1006 try 0 launched on executor multi with executor id 1007 parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/7b75866f-37f3-47da-a8ab-3d3b0af8c99a_isrForCrosstalkSources_1125903_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/7b75866f-37f3-47da-a8ab-3d3b0af8c99a_isrForCrosstalkSources_1125903_15.stderr parsl.dataflow.dflow DEBUG: Task 1007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1007 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1007 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1007 has memoization hash 39b42277c84e2665d1d2878be64323ce parsl.dataflow.memoization INFO: Task 1007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1007 try 0 launched on executor multi with executor id 1008 parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/fec609fc-4b32-4df8-9eaf-13a14a7fd88e_isrForCrosstalkSources_1125888_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/fec609fc-4b32-4df8-9eaf-13a14a7fd88e_isrForCrosstalkSources_1125888_52.stderr parsl.dataflow.dflow DEBUG: Task 1008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1008 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1008 has memoization hash b313f21d592599220208cf9fb370326f parsl.dataflow.memoization INFO: Task 1008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1008 try 0 launched on executor multi with executor id 1009 parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/fe47c397-fa70-485c-9ed4-f3b4dc23e573_isrForCrosstalkSources_1125935_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/fe47c397-fa70-485c-9ed4-f3b4dc23e573_isrForCrosstalkSources_1125935_16.stderr parsl.dataflow.dflow DEBUG: Task 1009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1009 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1009 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1009 has memoization hash 6c94d56b1f2b05a7cd8c65de4b30faaf parsl.dataflow.memoization INFO: Task 1009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1009 try 0 launched on executor multi with executor id 1010 parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/650676a5-4700-454f-9f9c-0e4901117e05_isrForCrosstalkSources_1125936_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/650676a5-4700-454f-9f9c-0e4901117e05_isrForCrosstalkSources_1125936_4.stderr parsl.dataflow.dflow DEBUG: Task 1010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1010 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1010 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1010 has memoization hash 5e3b2601d544b33d4659dfe59d057f3a parsl.dataflow.memoization INFO: Task 1010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d376a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1010 try 0 launched on executor multi with executor id 1011 parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/86ddf116-ffd3-4795-8834-4d4cb7aeabd1_isrForCrosstalkSources_1125924_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/86ddf116-ffd3-4795-8834-4d4cb7aeabd1_isrForCrosstalkSources_1125924_16.stderr parsl.dataflow.dflow DEBUG: Task 1011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1011 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1011 has memoization hash 4e93539a1541f0a0dbc62c0ccea82fd0 parsl.dataflow.memoization INFO: Task 1011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1011 try 0 launched on executor multi with executor id 1012 parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/ca3c059c-b787-4864-bc1f-219b17c4a54a_isrForCrosstalkSources_1125929_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/ca3c059c-b787-4864-bc1f-219b17c4a54a_isrForCrosstalkSources_1125929_12.stderr parsl.dataflow.dflow DEBUG: Task 1012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1012 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1012 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1012 has memoization hash 0ed93bced951a192ff1c8ba089b5e38b parsl.dataflow.memoization INFO: Task 1012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1012 try 0 launched on executor multi with executor id 1013 parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/1b1192ed-58d0-470c-886d-a04ad25ab038_isrForCrosstalkSources_1125910_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/1b1192ed-58d0-470c-886d-a04ad25ab038_isrForCrosstalkSources_1125910_42.stderr parsl.dataflow.dflow DEBUG: Task 1013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1013 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1013 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1013 has memoization hash f13787ae3c6e9cae93d984cb2ec13495 parsl.dataflow.memoization INFO: Task 1013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1013 try 0 launched on executor multi with executor id 1014 parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/15591753-1ab5-4809-abb6-1b5ca0768854_isrForCrosstalkSources_1125941_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/15591753-1ab5-4809-abb6-1b5ca0768854_isrForCrosstalkSources_1125941_53.stderr parsl.dataflow.dflow DEBUG: Task 1014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1014 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1014 has memoization hash ba2a19ca5f2c473b5b1c15c200f21a1c parsl.dataflow.memoization INFO: Task 1014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d379c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1014 try 0 launched on executor multi with executor id 1015 parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/0bf3e32a-2940-4bb2-ae61-363b6ac0ab0e_isrForCrosstalkSources_1125940_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/0bf3e32a-2940-4bb2-ae61-363b6ac0ab0e_isrForCrosstalkSources_1125940_54.stderr parsl.dataflow.dflow DEBUG: Task 1015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1015 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1015 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1015 has memoization hash c28cb287d2aad4ea23de4ca6cb69a355 parsl.dataflow.memoization INFO: Task 1015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d377e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1015 try 0 launched on executor multi with executor id 1016 parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/dea87cae-dbcf-4d98-9673-1d1656ffdbac_isrForCrosstalkSources_1125931_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/dea87cae-dbcf-4d98-9673-1d1656ffdbac_isrForCrosstalkSources_1125931_55.stderr parsl.dataflow.dflow DEBUG: Task 1016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1016 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1016 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1016 has memoization hash 973198b2fd74aa7cfdc1c23d028b085d parsl.dataflow.memoization INFO: Task 1016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1016 try 0 launched on executor multi with executor id 1017 parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/aa5ffa79-38dd-4133-9d19-d0e66ad3bee4_isrForCrosstalkSources_1125932_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/aa5ffa79-38dd-4133-9d19-d0e66ad3bee4_isrForCrosstalkSources_1125932_10.stderr parsl.dataflow.dflow DEBUG: Task 1017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1017 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1017 has memoization hash 6e25a6db8705bdb062e3da3c645ceeac parsl.dataflow.memoization INFO: Task 1017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1017 try 0 launched on executor multi with executor id 1018 parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/80ad6aaa-5f33-4f76-97b6-d74d4c3dacdb_isrForCrosstalkSources_1125888_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/80ad6aaa-5f33-4f76-97b6-d74d4c3dacdb_isrForCrosstalkSources_1125888_48.stderr parsl.dataflow.dflow DEBUG: Task 1018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1018 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1018 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1018 has memoization hash d4d845363459ea318e84c95e07d7cfe2 parsl.dataflow.memoization INFO: Task 1018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1018 try 0 launched on executor multi with executor id 1019 parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/559e49e8-3016-4cc2-bdb7-aa8d26718285_isrForCrosstalkSources_1125900_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/559e49e8-3016-4cc2-bdb7-aa8d26718285_isrForCrosstalkSources_1125900_61.stderr parsl.dataflow.dflow DEBUG: Task 1019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1019 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1019 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1019 has memoization hash 4275d42931a817dafe46978179e24263 parsl.dataflow.memoization INFO: Task 1019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1019 try 0 launched on executor multi with executor id 1020 parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/26ce8e7f-b70e-40c8-9b4c-5e4a57f12266_isrForCrosstalkSources_1125928_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/26ce8e7f-b70e-40c8-9b4c-5e4a57f12266_isrForCrosstalkSources_1125928_25.stderr parsl.dataflow.dflow DEBUG: Task 1020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1020 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1020 has memoization hash 3f05b49ac1115e112ff1d532a71bf685 parsl.dataflow.memoization INFO: Task 1020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1020 try 0 launched on executor multi with executor id 1021 parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/0392afca-97a8-4a72-bb3e-dd32823b5521_isrForCrosstalkSources_1125932_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/0392afca-97a8-4a72-bb3e-dd32823b5521_isrForCrosstalkSources_1125932_34.stderr parsl.dataflow.dflow DEBUG: Task 1021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1021 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1021 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1021 has memoization hash 4683ee44f7831f7461f5fc5426422cb9 parsl.dataflow.memoization INFO: Task 1021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1021 try 0 launched on executor multi with executor id 1022 parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ed700a1f-dab7-4d72-a4d9-6d296281dce7_isrForCrosstalkSources_1125894_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ed700a1f-dab7-4d72-a4d9-6d296281dce7_isrForCrosstalkSources_1125894_36.stderr parsl.dataflow.dflow DEBUG: Task 1022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1022 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1022 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1022 has memoization hash 936f20d467f0fa810bf08328633be110 parsl.dataflow.memoization INFO: Task 1022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1022 try 0 launched on executor multi with executor id 1023 parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/76132605-843e-4f65-8d05-5d0cf323cd70_isrForCrosstalkSources_1125942_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/76132605-843e-4f65-8d05-5d0cf323cd70_isrForCrosstalkSources_1125942_26.stderr parsl.dataflow.dflow DEBUG: Task 1023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1023 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1023 has memoization hash 99464f62c3f079c6613ed566a2ff4314 parsl.dataflow.memoization INFO: Task 1023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1023 try 0 launched on executor multi with executor id 1024 parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/0b70f8d2-d877-4a55-abc0-f94e2dcb160c_isrForCrosstalkSources_1125889_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/0b70f8d2-d877-4a55-abc0-f94e2dcb160c_isrForCrosstalkSources_1125889_31.stderr parsl.dataflow.dflow DEBUG: Task 1024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1024 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1024 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1024 has memoization hash 546be1f93007b92f32e4accbcf562091 parsl.dataflow.memoization INFO: Task 1024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:44 proc_lsst.multi:170 [INFO] [multi] job_id 16896396 proc_lsst.multi INFO: [multi] job_id 16896396 2024-03-07 15:06:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 15:06:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896396 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896396 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16896396 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-07 15:06:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-07 15:06:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 15:06:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1024 try 0 launched on executor multi with executor id 1025 parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/8b8f944e-e034-49aa-99e9-a82160db5b47_isrForCrosstalkSources_1125942_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/8b8f944e-e034-49aa-99e9-a82160db5b47_isrForCrosstalkSources_1125942_24.stderr parsl.dataflow.dflow DEBUG: Task 1025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1025 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1025 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1025 has memoization hash ccf2b41877aeb56f19e4dcb12c828003 parsl.dataflow.memoization INFO: Task 1025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d37100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1025 try 0 launched on executor multi with executor id 1026 parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ffa58167-38a6-4366-9001-16c975bc2d4d_isrForCrosstalkSources_1125894_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ffa58167-38a6-4366-9001-16c975bc2d4d_isrForCrosstalkSources_1125894_57.stderr parsl.dataflow.dflow DEBUG: Task 1026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1026 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1026 has memoization hash 7286a1788b6de484b9cf8e53be7ae150 parsl.dataflow.memoization INFO: Task 1026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1026 try 0 launched on executor multi with executor id 1027 parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/d760895a-0058-4060-b2de-6102083d4587_isrForCrosstalkSources_1125912_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/d760895a-0058-4060-b2de-6102083d4587_isrForCrosstalkSources_1125912_37.stderr parsl.dataflow.dflow DEBUG: Task 1027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1027 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1027 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1027 has memoization hash 5effd61836a01c8531e045fd230b1846 parsl.dataflow.memoization INFO: Task 1027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1027 try 0 launched on executor multi with executor id 1028 parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/266de6d8-4567-4f02-bfbb-a16b1c61b0a5_isrForCrosstalkSources_1125944_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/266de6d8-4567-4f02-bfbb-a16b1c61b0a5_isrForCrosstalkSources_1125944_36.stderr parsl.dataflow.dflow DEBUG: Task 1028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1028 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1028 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1028 has memoization hash 90753788753da8d373682f37efab18be parsl.dataflow.memoization INFO: Task 1028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1028 try 0 launched on executor multi with executor id 1029 parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/31e4839e-63ff-4d87-94af-56a793d39c7d_isrForCrosstalkSources_1125891_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/31e4839e-63ff-4d87-94af-56a793d39c7d_isrForCrosstalkSources_1125891_43.stderr parsl.dataflow.dflow DEBUG: Task 1029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1029 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1029 has memoization hash 1f413fe49e47e68c26d29f617eb7402d parsl.dataflow.memoization INFO: Task 1029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1029 try 0 launched on executor multi with executor id 1030 parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/a4c17cdd-abe5-41bb-af05-2492db2afac1_isrForCrosstalkSources_1125916_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/a4c17cdd-abe5-41bb-af05-2492db2afac1_isrForCrosstalkSources_1125916_36.stderr parsl.dataflow.dflow DEBUG: Task 1030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1030 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1030 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1030 has memoization hash 94147e8fc5e0f5522a56e5f24d23eb79 parsl.dataflow.memoization INFO: Task 1030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a47d36de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1030 try 0 launched on executor multi with executor id 1031 parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/31059be7-e114-4971-80cd-35a255d0dd32_isrForCrosstalkSources_1125936_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/31059be7-e114-4971-80cd-35a255d0dd32_isrForCrosstalkSources_1125936_61.stderr parsl.dataflow.dflow DEBUG: Task 1031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1031 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1031 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1031 has memoization hash b52aeb3b7979c9df62d112bfa27a0408 parsl.dataflow.memoization INFO: Task 1031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0840e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1031 try 0 launched on executor multi with executor id 1032 parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a1fdd4b4-5152-42d6-be3c-3d1eb1c891ee_isrForCrosstalkSources_1125900_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a1fdd4b4-5152-42d6-be3c-3d1eb1c891ee_isrForCrosstalkSources_1125900_43.stderr parsl.dataflow.dflow DEBUG: Task 1032 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1032 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1032 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1032 has memoization hash c66650b46c51b388c049b573ef9ae35c parsl.dataflow.memoization INFO: Task 1032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1032 try 0 launched on executor multi with executor id 1033 parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/00619d1f-c4f0-47d2-b5dc-8dfe850cea37_isrForCrosstalkSources_1125943_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/00619d1f-c4f0-47d2-b5dc-8dfe850cea37_isrForCrosstalkSources_1125943_37.stderr parsl.dataflow.dflow DEBUG: Task 1033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1033 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1033 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1033 has memoization hash 613812839b72a985511b1a1b3a7ee6fc parsl.dataflow.memoization INFO: Task 1033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1033 try 0 launched on executor multi with executor id 1034 parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/a4862c0d-ec44-44b0-82a1-7a5d3d0e6320_isrForCrosstalkSources_1125901_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/a4862c0d-ec44-44b0-82a1-7a5d3d0e6320_isrForCrosstalkSources_1125901_12.stderr parsl.dataflow.dflow DEBUG: Task 1034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1034 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1034 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1034 has memoization hash ea2c6a2e547ad91ff01776e159ba7831 parsl.dataflow.memoization INFO: Task 1034 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0877e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1034 try 0 launched on executor multi with executor id 1035 parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/e45b10ec-e384-4ef4-8b03-fe0eaafcd0f2_isrForCrosstalkSources_1125911_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/e45b10ec-e384-4ef4-8b03-fe0eaafcd0f2_isrForCrosstalkSources_1125911_24.stderr parsl.dataflow.dflow DEBUG: Task 1035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1035 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1035 has memoization hash 2d02054b28933ec602c7f0a840f6c9ca parsl.dataflow.memoization INFO: Task 1035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1035 try 0 launched on executor multi with executor id 1036 parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9d571f5e-0b55-4432-89e2-d3fa77210717_isrForCrosstalkSources_1125934_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9d571f5e-0b55-4432-89e2-d3fa77210717_isrForCrosstalkSources_1125934_41.stderr parsl.dataflow.dflow DEBUG: Task 1036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1036 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1036 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1036 has memoization hash 38276af9041f2f9e7e7dd2a4a8ea403b parsl.dataflow.memoization INFO: Task 1036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1036 try 0 launched on executor multi with executor id 1037 parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/c3356a0f-c7f9-474f-99f5-164705ce4e27_isrForCrosstalkSources_1125919_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/c3356a0f-c7f9-474f-99f5-164705ce4e27_isrForCrosstalkSources_1125919_57.stderr parsl.dataflow.dflow DEBUG: Task 1037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1037 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1037 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1037 has memoization hash 32f07d295499213e5d6f06194e98cdda parsl.dataflow.memoization INFO: Task 1037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0876a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1037 try 0 launched on executor multi with executor id 1038 parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/24cdd39f-d1e4-4db5-8d57-835145341b1f_isrForCrosstalkSources_1125908_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/24cdd39f-d1e4-4db5-8d57-835145341b1f_isrForCrosstalkSources_1125908_25.stderr parsl.dataflow.dflow DEBUG: Task 1038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1038 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1038 has memoization hash 17b1d3ca581dde71857ed1e9d7496a8b parsl.dataflow.memoization INFO: Task 1038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1038 try 0 launched on executor multi with executor id 1039 parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/90342246-beca-4e8a-9c08-5c1ca4d3945c_isrForCrosstalkSources_1125914_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/90342246-beca-4e8a-9c08-5c1ca4d3945c_isrForCrosstalkSources_1125914_45.stderr parsl.dataflow.dflow DEBUG: Task 1039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1039 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1039 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1039 has memoization hash f36f413645598385746511f66315d01f parsl.dataflow.memoization INFO: Task 1039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1039 try 0 launched on executor multi with executor id 1040 parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/3226aba0-7e13-40ee-9087-fa70680b8154_isrForCrosstalkSources_1125900_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/3226aba0-7e13-40ee-9087-fa70680b8154_isrForCrosstalkSources_1125900_53.stderr parsl.dataflow.dflow DEBUG: Task 1040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1040 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1040 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1040 has memoization hash 471d41f8e8b3722ba8cfd0895e02bb5a parsl.dataflow.memoization INFO: Task 1040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0847c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1040 try 0 launched on executor multi with executor id 1041 parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/765ea4b6-1d08-4e47-9f10-6a203b8d767e_isrForCrosstalkSources_1125930_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/765ea4b6-1d08-4e47-9f10-6a203b8d767e_isrForCrosstalkSources_1125930_47.stderr parsl.dataflow.dflow DEBUG: Task 1041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1041 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1041 has memoization hash 405f078a4df3d52dd39a1cbf8bbf5779 parsl.dataflow.memoization INFO: Task 1041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1041 try 0 launched on executor multi with executor id 1042 parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/3d1a00ab-8800-4792-aab2-df9dc8a22c35_isrForCrosstalkSources_1125906_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/3d1a00ab-8800-4792-aab2-df9dc8a22c35_isrForCrosstalkSources_1125906_55.stderr parsl.dataflow.dflow DEBUG: Task 1042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1042 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1042 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1042 has memoization hash 0075b62e3df8f1969249bc77350623bd parsl.dataflow.memoization INFO: Task 1042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1042 try 0 launched on executor multi with executor id 1043 parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/fe508941-b529-4924-95e9-f93cc5029f71_isrForCrosstalkSources_1125891_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/fe508941-b529-4924-95e9-f93cc5029f71_isrForCrosstalkSources_1125891_11.stderr parsl.dataflow.dflow DEBUG: Task 1043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1043 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1043 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1043 has memoization hash d36ce9f6aa5e2be11fc139fb296d0844 parsl.dataflow.memoization INFO: Task 1043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1043 try 0 launched on executor multi with executor id 1044 parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/c608ac50-f7c7-4608-9474-34cb7c3e3865_isrForCrosstalkSources_1125898_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/c608ac50-f7c7-4608-9474-34cb7c3e3865_isrForCrosstalkSources_1125898_56.stderr parsl.dataflow.dflow DEBUG: Task 1044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1044 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1044 has memoization hash 57faffa896964f38a03435a0330d3018 parsl.dataflow.memoization INFO: Task 1044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0849a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1044 try 0 launched on executor multi with executor id 1045 parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/52b8ae9a-2181-4f6a-bea0-bb32592ded73_isrForCrosstalkSources_1125914_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/52b8ae9a-2181-4f6a-bea0-bb32592ded73_isrForCrosstalkSources_1125914_21.stderr parsl.dataflow.dflow DEBUG: Task 1045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1045 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1045 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1045 has memoization hash 2ec406b93354fd178124887715cc13a3 parsl.dataflow.memoization INFO: Task 1045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1045 try 0 launched on executor multi with executor id 1046 parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/2dfd1ab9-9787-449c-836e-ff5db6a483f2_isrForCrosstalkSources_1125894_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/2dfd1ab9-9787-449c-836e-ff5db6a483f2_isrForCrosstalkSources_1125894_11.stderr parsl.dataflow.dflow DEBUG: Task 1046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1046 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1046 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1046 has memoization hash 685171c77a7cd77880f4b990da5cfbdb parsl.dataflow.memoization INFO: Task 1046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1046 try 0 launched on executor multi with executor id 1047 parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/c22b71a1-4649-4e0e-a5b9-a92cfb78b83e_isrForCrosstalkSources_1125934_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/c22b71a1-4649-4e0e-a5b9-a92cfb78b83e_isrForCrosstalkSources_1125934_36.stderr parsl.dataflow.dflow DEBUG: Task 1047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1047 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1047 has memoization hash 8260073bc9043b4a49729e8bcb232d6d parsl.dataflow.memoization INFO: Task 1047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1047 try 0 launched on executor multi with executor id 1048 parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/39a6c9b4-39a6-431a-b892-447a02af3000_isrForCrosstalkSources_1125928_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/39a6c9b4-39a6-431a-b892-447a02af3000_isrForCrosstalkSources_1125928_14.stderr parsl.dataflow.dflow DEBUG: Task 1048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1048 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1048 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1048 has memoization hash 899952601a4c222bb4f1a10e87cc97b0 parsl.dataflow.memoization INFO: Task 1048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1048 try 0 launched on executor multi with executor id 1049 parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/2f64299a-2878-4ca1-978b-138521d143cf_isrForCrosstalkSources_1125922_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/2f64299a-2878-4ca1-978b-138521d143cf_isrForCrosstalkSources_1125922_61.stderr parsl.dataflow.dflow DEBUG: Task 1049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1049 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1049 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1049 has memoization hash 9b912e3d063e65178f3430679186c665 parsl.dataflow.memoization INFO: Task 1049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1049 try 0 launched on executor multi with executor id 1050 parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/05e20cac-57ac-4f58-b9ce-f30246b88e25_isrForCrosstalkSources_1125901_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/05e20cac-57ac-4f58-b9ce-f30246b88e25_isrForCrosstalkSources_1125901_22.stderr parsl.dataflow.dflow DEBUG: Task 1050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1050 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1050 has memoization hash 485abb6f70e8330ec46380284ed9c8f8 parsl.dataflow.memoization INFO: Task 1050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1050 try 0 launched on executor multi with executor id 1051 parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/9cdab96c-d9fb-47b0-94d9-872babda8b42_isrForCrosstalkSources_1125899_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/9cdab96c-d9fb-47b0-94d9-872babda8b42_isrForCrosstalkSources_1125899_22.stderr parsl.dataflow.dflow DEBUG: Task 1051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1051 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1051 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1051 has memoization hash 3af9fd7ddc78631037eaf24f601eb1f2 parsl.dataflow.memoization INFO: Task 1051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1051 try 0 launched on executor multi with executor id 1052 parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/0be30802-0544-4db0-ab13-76db154c3b64_isrForCrosstalkSources_1125908_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/0be30802-0544-4db0-ab13-76db154c3b64_isrForCrosstalkSources_1125908_9.stderr parsl.dataflow.dflow DEBUG: Task 1052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1052 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1052 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1052 has memoization hash 0de9da140a64c94793cb41d5f279bbc8 parsl.dataflow.memoization INFO: Task 1052 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1052 try 0 launched on executor multi with executor id 1053 parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/71dcf546-367b-4827-856d-618e9b624797_isrForCrosstalkSources_1125904_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/71dcf546-367b-4827-856d-618e9b624797_isrForCrosstalkSources_1125904_1.stderr parsl.dataflow.dflow DEBUG: Task 1053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1053 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1053 has memoization hash f69e79d0da647c97c605bc1c0ac58ddf parsl.dataflow.memoization INFO: Task 1053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1053 try 0 launched on executor multi with executor id 1054 parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/a63c9ee1-9d08-48be-ade3-230a7485e2d2_isrForCrosstalkSources_1125896_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/a63c9ee1-9d08-48be-ade3-230a7485e2d2_isrForCrosstalkSources_1125896_42.stderr parsl.dataflow.dflow DEBUG: Task 1054 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1054 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1054 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1054 has memoization hash a8ed4f294f12a1a4e880b77d603188d9 parsl.dataflow.memoization INFO: Task 1054 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1054 try 0 launched on executor multi with executor id 1055 parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/ba7bd594-f745-4bf7-a1f3-ffe1367c4907_isrForCrosstalkSources_1125898_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/ba7bd594-f745-4bf7-a1f3-ffe1367c4907_isrForCrosstalkSources_1125898_25.stderr parsl.dataflow.dflow DEBUG: Task 1055 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1055 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1055 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1055 has memoization hash 1ae14e7cfd7e7cb57f9794a22b4cb851 parsl.dataflow.memoization INFO: Task 1055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1055 try 0 launched on executor multi with executor id 1056 parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/d7f57580-01a9-450e-8bc9-7c8a95e1e048_isrForCrosstalkSources_1125914_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/d7f57580-01a9-450e-8bc9-7c8a95e1e048_isrForCrosstalkSources_1125914_48.stderr parsl.dataflow.dflow DEBUG: Task 1056 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1056 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1056 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1056 has memoization hash 14ca390ed141a167eae79831aa502da5 parsl.dataflow.memoization INFO: Task 1056 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1056 try 0 launched on executor multi with executor id 1057 parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/3b75714f-7442-4ceb-b63e-00631e68a188_isrForCrosstalkSources_1125903_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/3b75714f-7442-4ceb-b63e-00631e68a188_isrForCrosstalkSources_1125903_22.stderr parsl.dataflow.dflow DEBUG: Task 1057 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1057 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1057 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1057 has memoization hash 266d4e0df286633f21b8e20565c133d7 parsl.dataflow.memoization INFO: Task 1057 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1057 try 0 launched on executor multi with executor id 1058 parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/e1baa4cb-2b3b-4701-9362-180e362d70ae_isrForCrosstalkSources_1125889_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/e1baa4cb-2b3b-4701-9362-180e362d70ae_isrForCrosstalkSources_1125889_17.stderr parsl.dataflow.dflow DEBUG: Task 1058 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1058 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1058 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1058 has memoization hash 1044cb01bf7e7c309c3fd75009f21b4f parsl.dataflow.memoization INFO: Task 1058 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0851c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1058 try 0 launched on executor multi with executor id 1059 parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/d8af5b33-59eb-4952-a1b2-4d55bc82de2c_isrForCrosstalkSources_1125936_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/d8af5b33-59eb-4952-a1b2-4d55bc82de2c_isrForCrosstalkSources_1125936_59.stderr parsl.dataflow.dflow DEBUG: Task 1059 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1059 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1059 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1059 has memoization hash 8ff335a06c32b06f5f1f4edd19583433 parsl.dataflow.memoization INFO: Task 1059 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0853a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1059 try 0 launched on executor multi with executor id 1060 parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/d86c36b0-b986-4975-94a2-b54d921b3d17_isrForCrosstalkSources_1125899_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/d86c36b0-b986-4975-94a2-b54d921b3d17_isrForCrosstalkSources_1125899_47.stderr parsl.dataflow.dflow DEBUG: Task 1060 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1060 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1060 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1060 has memoization hash ae61cbe4a6b934f7729f179528a5bb56 parsl.dataflow.memoization INFO: Task 1060 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1060 try 0 launched on executor multi with executor id 1061 parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/65d8aa89-daa0-4e12-aa84-6721e904984c_isrForCrosstalkSources_1125903_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/65d8aa89-daa0-4e12-aa84-6721e904984c_isrForCrosstalkSources_1125903_2.stderr parsl.dataflow.dflow DEBUG: Task 1061 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1061 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1061 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1061 has memoization hash bc2db30ff24c952de72f8f08593ef76a parsl.dataflow.memoization INFO: Task 1061 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1061 try 0 launched on executor multi with executor id 1062 parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/da847968-87b1-4f8f-9146-04bd80950c48_isrForCrosstalkSources_1125933_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/da847968-87b1-4f8f-9146-04bd80950c48_isrForCrosstalkSources_1125933_61.stderr parsl.dataflow.dflow DEBUG: Task 1062 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1062 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1062 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1062 has memoization hash 26157f07a8de2135ab04a752187bbbdd parsl.dataflow.memoization INFO: Task 1062 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0854e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1062 try 0 launched on executor multi with executor id 1063 parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/7999f8e7-156d-414c-817c-f40001981e7a_isrForCrosstalkSources_1125945_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/7999f8e7-156d-414c-817c-f40001981e7a_isrForCrosstalkSources_1125945_41.stderr parsl.dataflow.dflow DEBUG: Task 1063 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1063 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1063 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1063 has memoization hash 03abd6cda47afb7a642f96b0d5ec1b88 parsl.dataflow.memoization INFO: Task 1063 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1063 try 0 launched on executor multi with executor id 1064 parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/0eef2f80-34da-43ea-98f7-13669ed4a9a7_isrForCrosstalkSources_1125898_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/0eef2f80-34da-43ea-98f7-13669ed4a9a7_isrForCrosstalkSources_1125898_47.stderr parsl.dataflow.dflow DEBUG: Task 1064 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1064 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1064 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1064 has memoization hash e24ba14724fae5a4f0ac276106555a39 parsl.dataflow.memoization INFO: Task 1064 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:45 proc_lsst.multi:170 [INFO] [multi] job_id 16896397 proc_lsst.multi INFO: [multi] job_id 16896397 2024-03-07 15:06:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 15:06:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896397 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896397 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16896397 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-07 15:06:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-07 15:06:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 15:06:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1064 try 0 launched on executor multi with executor id 1065 parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/52874518-ba3d-4752-a17a-c09e41b10e57_isrForCrosstalkSources_1125936_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/52874518-ba3d-4752-a17a-c09e41b10e57_isrForCrosstalkSources_1125936_17.stderr parsl.dataflow.dflow DEBUG: Task 1065 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1065 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1065 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1065 has memoization hash d380d70681697f48306406a69900609e parsl.dataflow.memoization INFO: Task 1065 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1065 try 0 launched on executor multi with executor id 1066 parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/726fe89b-52a8-4329-a9c9-1a5c0ca0ef98_isrForCrosstalkSources_1125912_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/726fe89b-52a8-4329-a9c9-1a5c0ca0ef98_isrForCrosstalkSources_1125912_35.stderr parsl.dataflow.dflow DEBUG: Task 1066 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1066 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1066 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1066 has memoization hash 066463ca4f694c2f62449f69c5c734b5 parsl.dataflow.memoization INFO: Task 1066 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0845e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1066 try 0 launched on executor multi with executor id 1067 parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/b2fdf050-07f9-492d-ac01-0db9a272df68_isrForCrosstalkSources_1125918_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/b2fdf050-07f9-492d-ac01-0db9a272df68_isrForCrosstalkSources_1125918_35.stderr parsl.dataflow.dflow DEBUG: Task 1067 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1067 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1067 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1067 has memoization hash cdd7a7010a40e265a02b99922f69907c parsl.dataflow.memoization INFO: Task 1067 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0844a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1067 try 0 launched on executor multi with executor id 1068 parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/9415764e-8a48-4c40-95a0-95f549ab86d8_isrForCrosstalkSources_1125946_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/9415764e-8a48-4c40-95a0-95f549ab86d8_isrForCrosstalkSources_1125946_4.stderr parsl.dataflow.dflow DEBUG: Task 1068 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1068 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1068 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1068 has memoization hash d243935427de07d83313ac1fbd85f2d2 parsl.dataflow.memoization INFO: Task 1068 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0858a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1068 try 0 launched on executor multi with executor id 1069 parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/271bfc21-930c-4916-867c-6221644ca047_isrForCrosstalkSources_1125930_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/271bfc21-930c-4916-867c-6221644ca047_isrForCrosstalkSources_1125930_48.stderr parsl.dataflow.dflow DEBUG: Task 1069 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1069 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1069 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1069 has memoization hash db35698dc76a89ead0036bbeef0af830 parsl.dataflow.memoization INFO: Task 1069 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1069 try 0 launched on executor multi with executor id 1070 parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/0ff861f6-fc01-416e-8ec5-82940fe21d8c_isrForCrosstalkSources_1125947_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/0ff861f6-fc01-416e-8ec5-82940fe21d8c_isrForCrosstalkSources_1125947_53.stderr parsl.dataflow.dflow DEBUG: Task 1070 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1070 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1070 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1070 has memoization hash d4180c204242866f85ef053479514cbe parsl.dataflow.memoization INFO: Task 1070 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1070 try 0 launched on executor multi with executor id 1071 parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/c6fb3f64-dae7-4f3a-91b0-1fd468d2ceff_isrForCrosstalkSources_1125894_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/c6fb3f64-dae7-4f3a-91b0-1fd468d2ceff_isrForCrosstalkSources_1125894_12.stderr parsl.dataflow.dflow DEBUG: Task 1071 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1071 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1071 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1071 has memoization hash 3d926c509ce224fa6fa4bf6b8d6f99d4 parsl.dataflow.memoization INFO: Task 1071 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0856c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1071 try 0 launched on executor multi with executor id 1072 parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/61dc49f3-87d8-4db5-ab17-1e82cf299b46_isrForCrosstalkSources_1125947_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/61dc49f3-87d8-4db5-ab17-1e82cf299b46_isrForCrosstalkSources_1125947_25.stderr parsl.dataflow.dflow DEBUG: Task 1072 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1072 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1072 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1072 has memoization hash 3004730c6e0fa113004e496a8c05cd0e parsl.dataflow.memoization INFO: Task 1072 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1072 try 0 launched on executor multi with executor id 1073 parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/fd5117ad-2b16-40bf-a1a2-e573cd8462cf_isrForCrosstalkSources_1125928_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/fd5117ad-2b16-40bf-a1a2-e573cd8462cf_isrForCrosstalkSources_1125928_2.stderr parsl.dataflow.dflow DEBUG: Task 1073 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1073 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1073 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1073 has memoization hash 834873a60fce5d326f05ababc231525c parsl.dataflow.memoization INFO: Task 1073 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1073 try 0 launched on executor multi with executor id 1074 parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/2fd19f65-ed17-481c-8305-e394105c681d_isrForCrosstalkSources_1125934_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/2fd19f65-ed17-481c-8305-e394105c681d_isrForCrosstalkSources_1125934_55.stderr parsl.dataflow.dflow DEBUG: Task 1074 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1074 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1074 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1074 has memoization hash 1053594514f0d0ace1b9552ff2f3ed22 parsl.dataflow.memoization INFO: Task 1074 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1074 try 0 launched on executor multi with executor id 1075 parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/d38623c8-99ca-49e7-9da9-a4962b28f99d_isrForCrosstalkSources_1125927_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/d38623c8-99ca-49e7-9da9-a4962b28f99d_isrForCrosstalkSources_1125927_62.stderr parsl.dataflow.dflow DEBUG: Task 1075 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1075 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1075 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1075 has memoization hash b136c3c1f36af802f7051dfd45a28e20 parsl.dataflow.memoization INFO: Task 1075 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1075 try 0 launched on executor multi with executor id 1076 parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/3116fd6d-0cce-493e-8453-b884dc2a9f0e_isrForCrosstalkSources_1125919_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/3116fd6d-0cce-493e-8453-b884dc2a9f0e_isrForCrosstalkSources_1125919_35.stderr parsl.dataflow.dflow DEBUG: Task 1076 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1076 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1076 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1076 has memoization hash b0800d10d46bffcb74781f8f7556c9da parsl.dataflow.memoization INFO: Task 1076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1076 try 0 launched on executor multi with executor id 1077 parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/a681c070-3292-4c67-82aa-b421e235e5a9_isrForCrosstalkSources_1125892_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/a681c070-3292-4c67-82aa-b421e235e5a9_isrForCrosstalkSources_1125892_45.stderr parsl.dataflow.dflow DEBUG: Task 1077 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1077 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1077 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1077 has memoization hash ef394c09d6ffae6e4f6ad3a14915a7c6 parsl.dataflow.memoization INFO: Task 1077 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1077 try 0 launched on executor multi with executor id 1078 parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a9854239-66c7-46ea-9dcc-5b74b6cb1196_isrForCrosstalkSources_1125921_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a9854239-66c7-46ea-9dcc-5b74b6cb1196_isrForCrosstalkSources_1125921_26.stderr parsl.dataflow.dflow DEBUG: Task 1078 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1078 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1078 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1078 has memoization hash 6aa420b4e7749d0b3e22cfd0e1acff72 parsl.dataflow.memoization INFO: Task 1078 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1078 try 0 launched on executor multi with executor id 1079 parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/7b98e072-eceb-458f-b9db-d037d135fc7e_isrForCrosstalkSources_1125897_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/7b98e072-eceb-458f-b9db-d037d135fc7e_isrForCrosstalkSources_1125897_56.stderr parsl.dataflow.dflow DEBUG: Task 1079 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1079 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1079 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1079 has memoization hash 172abe287accd7aae1c42e30ecc39f16 parsl.dataflow.memoization INFO: Task 1079 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1079 try 0 launched on executor multi with executor id 1080 parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/6a1f1274-db59-4b0a-a64e-b14e7b1d70ac_isrForCrosstalkSources_1125893_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/6a1f1274-db59-4b0a-a64e-b14e7b1d70ac_isrForCrosstalkSources_1125893_1.stderr parsl.dataflow.dflow DEBUG: Task 1080 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1080 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1080 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1080 has memoization hash 9fc3f2042559a5e5736da49d8fc898ad parsl.dataflow.memoization INFO: Task 1080 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1080 try 0 launched on executor multi with executor id 1081 parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/90080d39-7049-4768-a0c2-db2cdc742db4_isrForCrosstalkSources_1125920_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/90080d39-7049-4768-a0c2-db2cdc742db4_isrForCrosstalkSources_1125920_5.stderr parsl.dataflow.dflow DEBUG: Task 1081 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1081 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1081 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1081 has memoization hash 8f4cf70c635896b4fcb62ddb802248fa parsl.dataflow.memoization INFO: Task 1081 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0859e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1081 try 0 launched on executor multi with executor id 1082 parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/9f134e5a-ab64-4dee-b30d-d2cb40053981_isrForCrosstalkSources_1125924_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/9f134e5a-ab64-4dee-b30d-d2cb40053981_isrForCrosstalkSources_1125924_35.stderr parsl.dataflow.dflow DEBUG: Task 1082 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1082 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1082 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1082 has memoization hash f852bbe02c193508cf8101300360a612 parsl.dataflow.memoization INFO: Task 1082 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1082 try 0 launched on executor multi with executor id 1083 parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/f99a9ee3-7ae2-4cd5-ba0f-47a815a52235_isrForCrosstalkSources_1125928_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/f99a9ee3-7ae2-4cd5-ba0f-47a815a52235_isrForCrosstalkSources_1125928_46.stderr parsl.dataflow.dflow DEBUG: Task 1083 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1083 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1083 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1083 has memoization hash 4b17a08c238901724f20d31bd8c3edad parsl.dataflow.memoization INFO: Task 1083 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1083 try 0 launched on executor multi with executor id 1084 parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/be5a26ff-d00b-4210-b096-799ffec0986d_isrForCrosstalkSources_1125935_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/be5a26ff-d00b-4210-b096-799ffec0986d_isrForCrosstalkSources_1125935_53.stderr parsl.dataflow.dflow DEBUG: Task 1084 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1084 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1084 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1084 has memoization hash 044bf44dd6f24cc6b8607bc7c802348b parsl.dataflow.memoization INFO: Task 1084 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1084 try 0 launched on executor multi with executor id 1085 parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/778bd707-4a8d-44b4-b667-45cbf2726360_isrForCrosstalkSources_1125921_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/778bd707-4a8d-44b4-b667-45cbf2726360_isrForCrosstalkSources_1125921_21.stderr parsl.dataflow.dflow DEBUG: Task 1085 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1085 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1085 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1085 has memoization hash 923964ab20b6c6c9faf48c3a9fd787bc parsl.dataflow.memoization INFO: Task 1085 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1085 try 0 launched on executor multi with executor id 1086 parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/c1298692-0bd2-48bc-b454-d380e715c311_isrForCrosstalkSources_1125928_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/c1298692-0bd2-48bc-b454-d380e715c311_isrForCrosstalkSources_1125928_27.stderr parsl.dataflow.dflow DEBUG: Task 1086 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1086 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1086 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1086 has memoization hash 1f3b697db21dc78a8de354bc732e3759 parsl.dataflow.memoization INFO: Task 1086 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1086 try 0 launched on executor multi with executor id 1087 parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/c735aa07-1259-4f54-8472-28e0cbf2ce51_isrForCrosstalkSources_1125924_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/c735aa07-1259-4f54-8472-28e0cbf2ce51_isrForCrosstalkSources_1125924_13.stderr parsl.dataflow.dflow DEBUG: Task 1087 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1087 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1087 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1087 has memoization hash 419bd8bf458c5d900479dc013c710466 parsl.dataflow.memoization INFO: Task 1087 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1087 try 0 launched on executor multi with executor id 1088 parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/4dcc754d-e0d9-43cf-8d10-85cf4694ca97_isrForCrosstalkSources_1125943_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/4dcc754d-e0d9-43cf-8d10-85cf4694ca97_isrForCrosstalkSources_1125943_11.stderr parsl.dataflow.dflow DEBUG: Task 1088 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1088 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1088 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1088 has memoization hash 52320a385d9dc329db6064f6d4d2fdac parsl.dataflow.memoization INFO: Task 1088 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1088 try 0 launched on executor multi with executor id 1089 parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/0f7134ce-000a-46d9-9424-3774e96cf259_isrForCrosstalkSources_1125928_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/0f7134ce-000a-46d9-9424-3774e96cf259_isrForCrosstalkSources_1125928_11.stderr parsl.dataflow.dflow DEBUG: Task 1089 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1089 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1089 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1089 has memoization hash 78cf51e67ab4f7807551e7e403f53d1f parsl.dataflow.memoization INFO: Task 1089 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1089 try 0 launched on executor multi with executor id 1090 parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/af78e220-d447-471f-a044-6c1e73937002_isrForCrosstalkSources_1125939_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/af78e220-d447-471f-a044-6c1e73937002_isrForCrosstalkSources_1125939_25.stderr parsl.dataflow.dflow DEBUG: Task 1090 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1090 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1090 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1090 has memoization hash 09e98d770058e06916a0f3ad40209ab1 parsl.dataflow.memoization INFO: Task 1090 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0862a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1090 try 0 launched on executor multi with executor id 1091 parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/9ee296eb-7a42-4528-a1e2-c5536a9ddfbf_isrForCrosstalkSources_1125942_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/9ee296eb-7a42-4528-a1e2-c5536a9ddfbf_isrForCrosstalkSources_1125942_2.stderr parsl.dataflow.dflow DEBUG: Task 1091 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1091 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1091 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1091 has memoization hash 35211293074c478ee36528331baef72f parsl.dataflow.memoization INFO: Task 1091 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1091 try 0 launched on executor multi with executor id 1092 parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/98b57a0e-49f0-402c-89aa-f0851f4447a6_isrForCrosstalkSources_1125906_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/98b57a0e-49f0-402c-89aa-f0851f4447a6_isrForCrosstalkSources_1125906_16.stderr parsl.dataflow.dflow DEBUG: Task 1092 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1092 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1092 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1092 has memoization hash cbe4e9b6e5d98a4916e086461e996194 parsl.dataflow.memoization INFO: Task 1092 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1092 try 0 launched on executor multi with executor id 1093 parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/75fd488c-2437-4778-96d2-00d8ec506d4e_isrForCrosstalkSources_1125944_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/75fd488c-2437-4778-96d2-00d8ec506d4e_isrForCrosstalkSources_1125944_60.stderr parsl.dataflow.dflow DEBUG: Task 1093 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1093 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1093 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1093 has memoization hash 344c51f96587c1fefb499bc808f8b1a5 parsl.dataflow.memoization INFO: Task 1093 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0863e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1093 try 0 launched on executor multi with executor id 1094 parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/07ac3019-2b72-445e-adce-3a108b816641_isrForCrosstalkSources_1125941_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/07ac3019-2b72-445e-adce-3a108b816641_isrForCrosstalkSources_1125941_4.stderr parsl.dataflow.dflow DEBUG: Task 1094 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1094 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1094 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1094 has memoization hash 5d1f3a50e6a47087836e8548563edb10 parsl.dataflow.memoization INFO: Task 1094 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0860c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1094 try 0 launched on executor multi with executor id 1095 parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a29f6450-1edc-4a88-9638-19317a3e94cc_isrForCrosstalkSources_1125945_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a29f6450-1edc-4a88-9638-19317a3e94cc_isrForCrosstalkSources_1125945_16.stderr parsl.dataflow.dflow DEBUG: Task 1095 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1095 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1095 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1095 has memoization hash 02a9632cc2e9924bb08072176e054f7b parsl.dataflow.memoization INFO: Task 1095 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1095 try 0 launched on executor multi with executor id 1096 parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/4c8749d6-b0e6-4650-9963-c6269563b6a3_isrForCrosstalkSources_1125944_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/4c8749d6-b0e6-4650-9963-c6269563b6a3_isrForCrosstalkSources_1125944_39.stderr parsl.dataflow.dflow DEBUG: Task 1096 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1096 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1096 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1096 has memoization hash 591f98b82ebe9395bde6ce7f734c5de9 parsl.dataflow.memoization INFO: Task 1096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1096 try 0 launched on executor multi with executor id 1097 parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/eea7a7f4-f814-4a31-9b56-b7ad91c8f776_isrForCrosstalkSources_1125921_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/eea7a7f4-f814-4a31-9b56-b7ad91c8f776_isrForCrosstalkSources_1125921_9.stderr parsl.dataflow.dflow DEBUG: Task 1097 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1097 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1097 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1097 has memoization hash 8dd3bdecb317f23ee08ab3f4b4e5fb06 parsl.dataflow.memoization INFO: Task 1097 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0872e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1097 try 0 launched on executor multi with executor id 1098 parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/973ebdb1-2233-463a-9004-916d391d3747_isrForCrosstalkSources_1125904_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/973ebdb1-2233-463a-9004-916d391d3747_isrForCrosstalkSources_1125904_43.stderr parsl.dataflow.dflow DEBUG: Task 1098 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1098 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1098 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1098 has memoization hash 95dcb6c0ffbdfde7ea68dc0a466c95b7 parsl.dataflow.memoization INFO: Task 1098 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1098 try 0 launched on executor multi with executor id 1099 parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/f2c334e1-df4c-4483-87e9-b511413644ee_isrForCrosstalkSources_1125925_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/f2c334e1-df4c-4483-87e9-b511413644ee_isrForCrosstalkSources_1125925_41.stderr parsl.dataflow.dflow DEBUG: Task 1099 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1099 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1099 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1099 has memoization hash 7323b04c9f599e91870d6cf57443c763 parsl.dataflow.memoization INFO: Task 1099 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0874c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1099 try 0 launched on executor multi with executor id 1100 parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/2d7abb00-11c3-486b-b206-6d0819e185e0_isrForCrosstalkSources_1125901_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/2d7abb00-11c3-486b-b206-6d0819e185e0_isrForCrosstalkSources_1125901_54.stderr parsl.dataflow.dflow DEBUG: Task 1100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1100 has memoization hash 375e9e6028e37458b0f34dd7a361d10f parsl.dataflow.memoization INFO: Task 1100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0871a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1100 try 0 launched on executor multi with executor id 1101 parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/fac629ea-a6a3-476a-ae2a-c7ddaf08edb5_isrForCrosstalkSources_1125895_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/fac629ea-a6a3-476a-ae2a-c7ddaf08edb5_isrForCrosstalkSources_1125895_45.stderr parsl.dataflow.dflow DEBUG: Task 1101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1101 has memoization hash 491f769b46d76b2a72daa382af4288a3 parsl.dataflow.memoization INFO: Task 1101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1101 try 0 launched on executor multi with executor id 1102 parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/02da22bb-976e-4992-bc5d-62918018fcb5_isrForCrosstalkSources_1125919_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/02da22bb-976e-4992-bc5d-62918018fcb5_isrForCrosstalkSources_1125919_2.stderr parsl.dataflow.dflow DEBUG: Task 1102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1102 has memoization hash 8b8d52ef69524f601ce77f84132f0b15 parsl.dataflow.memoization INFO: Task 1102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1102 try 0 launched on executor multi with executor id 1103 parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2a83470b-c7ab-4c01-9821-417c7fa86938_isrForCrosstalkSources_1125898_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2a83470b-c7ab-4c01-9821-417c7fa86938_isrForCrosstalkSources_1125898_42.stderr parsl.dataflow.dflow DEBUG: Task 1103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1103 has memoization hash 5443cf54874fc437cbd925777f6fe8c4 parsl.dataflow.memoization INFO: Task 1103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1103 try 0 launched on executor multi with executor id 1104 parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/0c25c76a-9d22-4cfb-b77b-f76118fa2172_isrForCrosstalkSources_1125934_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/0c25c76a-9d22-4cfb-b77b-f76118fa2172_isrForCrosstalkSources_1125934_54.stderr parsl.dataflow.dflow DEBUG: Task 1104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1104 has memoization hash 0dd857224e67763bada7f5a35d7ad4c8 parsl.dataflow.memoization INFO: Task 1104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1104 try 0 launched on executor multi with executor id 1105 parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/d564c2cf-ad62-48be-bd75-71b038c8e329_isrForCrosstalkSources_1125900_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/d564c2cf-ad62-48be-bd75-71b038c8e329_isrForCrosstalkSources_1125900_45.stderr parsl.dataflow.dflow DEBUG: Task 1105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1105 has memoization hash 7b92403ae3ccc48819262f6c0d5113e0 parsl.dataflow.memoization INFO: Task 1105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1105 try 0 launched on executor multi with executor id 1106 parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/88f437b2-4e14-40e4-9252-2d3e2f786210_isrForCrosstalkSources_1125921_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/88f437b2-4e14-40e4-9252-2d3e2f786210_isrForCrosstalkSources_1125921_62.stderr parsl.dataflow.dflow DEBUG: Task 1106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1106 has memoization hash c4a52ddccaca5ea3ef6bebdd44f25cba parsl.dataflow.memoization INFO: Task 1106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1106 try 0 launched on executor multi with executor id 1107 parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/76644ed0-586a-4e43-861a-3f1cff5ad722_isrForCrosstalkSources_1125893_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/76644ed0-586a-4e43-861a-3f1cff5ad722_isrForCrosstalkSources_1125893_43.stderr parsl.dataflow.dflow DEBUG: Task 1107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1107 has memoization hash 500b3e302decdd16b1f594184cff96f1 parsl.dataflow.memoization INFO: Task 1107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1107 try 0 launched on executor multi with executor id 1108 parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/7f4ddf15-1349-4b3b-a0e6-9c71078eef61_isrForCrosstalkSources_1125918_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/7f4ddf15-1349-4b3b-a0e6-9c71078eef61_isrForCrosstalkSources_1125918_50.stderr parsl.dataflow.dflow DEBUG: Task 1108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1108 has memoization hash 5d7dd0f880d2c160d241119595bccbac parsl.dataflow.memoization INFO: Task 1108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1108 try 0 launched on executor multi with executor id 1109 parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/535fa8e8-e801-4e0e-ad14-bf8b9a3a18d8_isrForCrosstalkSources_1125940_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/535fa8e8-e801-4e0e-ad14-bf8b9a3a18d8_isrForCrosstalkSources_1125940_14.stderr parsl.dataflow.dflow DEBUG: Task 1109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1109 has memoization hash 55546615af048442a502277acbd6c881 parsl.dataflow.memoization INFO: Task 1109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1109 try 0 launched on executor multi with executor id 1110 parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/29d06600-2000-442a-a992-a32e5e1ba806_isrForCrosstalkSources_1125943_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/29d06600-2000-442a-a992-a32e5e1ba806_isrForCrosstalkSources_1125943_56.stderr parsl.dataflow.dflow DEBUG: Task 1110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1110 has memoization hash f4e214153fbe4f86b2d21436d4916389 parsl.dataflow.memoization INFO: Task 1110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1110 try 0 launched on executor multi with executor id 1111 parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/574b9e96-8ce9-452d-9de3-10ce93445fe3_isrForCrosstalkSources_1125923_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/574b9e96-8ce9-452d-9de3-10ce93445fe3_isrForCrosstalkSources_1125923_58.stderr parsl.dataflow.dflow DEBUG: Task 1111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1111 has memoization hash 7200f3c2d019c54d7442e0db001f001d parsl.dataflow.memoization INFO: Task 1111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1111 try 0 launched on executor multi with executor id 1112 parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/45eedce9-eb95-4e9f-aec5-b0e221275458_isrForCrosstalkSources_1125943_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/45eedce9-eb95-4e9f-aec5-b0e221275458_isrForCrosstalkSources_1125943_30.stderr parsl.dataflow.dflow DEBUG: Task 1112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1112 has memoization hash 7d4893f2d6c09ab3a1722ae15423c0f9 parsl.dataflow.memoization INFO: Task 1112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1112 try 0 launched on executor multi with executor id 1113 parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/4cd57217-d65b-49a1-8fa3-f36181d0dc4c_isrForCrosstalkSources_1125895_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/4cd57217-d65b-49a1-8fa3-f36181d0dc4c_isrForCrosstalkSources_1125895_60.stderr parsl.dataflow.dflow DEBUG: Task 1113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1113 has memoization hash c34c74d64f41ec58e0a83fb76319b643 parsl.dataflow.memoization INFO: Task 1113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1113 try 0 launched on executor multi with executor id 1114 parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/6fe3f8df-c3ca-435d-9550-61182a732e64_isrForCrosstalkSources_1125890_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/6fe3f8df-c3ca-435d-9550-61182a732e64_isrForCrosstalkSources_1125890_46.stderr parsl.dataflow.dflow DEBUG: Task 1114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1114 has memoization hash f7bbb2b779a3b2ceb9c7fd7b4960c42b parsl.dataflow.memoization INFO: Task 1114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1114 try 0 launched on executor multi with executor id 1115 parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/882cabcb-58e7-40a5-b906-7fe7f47a8425_isrForCrosstalkSources_1125897_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/882cabcb-58e7-40a5-b906-7fe7f47a8425_isrForCrosstalkSources_1125897_29.stderr parsl.dataflow.dflow DEBUG: Task 1115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1115 has memoization hash bec2b91845515f70d4bbef53eb811b1a parsl.dataflow.memoization INFO: Task 1115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1115 try 0 launched on executor multi with executor id 1116 parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/60a08456-5ed6-411f-b5ae-2115f89ae05f_isrForCrosstalkSources_1125942_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/60a08456-5ed6-411f-b5ae-2115f89ae05f_isrForCrosstalkSources_1125942_47.stderr parsl.dataflow.dflow DEBUG: Task 1116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1116 has memoization hash 9e5e19b3854d6e433590472d2f6cf095 parsl.dataflow.memoization INFO: Task 1116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0867a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1116 try 0 launched on executor multi with executor id 1117 parsl.dataflow.dflow INFO: Standard output for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/5b4e42e2-58c6-4f39-acf6-929077d40f92_isrForCrosstalkSources_1125925_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/5b4e42e2-58c6-4f39-acf6-929077d40f92_isrForCrosstalkSources_1125925_47.stderr parsl.dataflow.dflow DEBUG: Task 1117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1117 has memoization hash c6307a69ed81f5483bb52de6328628f0 parsl.dataflow.memoization INFO: Task 1117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0868e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1117 try 0 launched on executor multi with executor id 1118 parsl.dataflow.dflow INFO: Standard output for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/6e7e5217-d9de-41bc-935c-aa8414487b7a_isrForCrosstalkSources_1125943_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/6e7e5217-d9de-41bc-935c-aa8414487b7a_isrForCrosstalkSources_1125943_21.stderr parsl.dataflow.dflow DEBUG: Task 1118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1118 has memoization hash 50221b88eb4f554b976bb8cf8a30bfd6 parsl.dataflow.memoization INFO: Task 1118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1118 try 0 launched on executor multi with executor id 1119 parsl.dataflow.dflow INFO: Standard output for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/ee8a0452-9cad-42e2-abee-cfe30571d229_isrForCrosstalkSources_1125938_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/ee8a0452-9cad-42e2-abee-cfe30571d229_isrForCrosstalkSources_1125938_62.stderr parsl.dataflow.dflow DEBUG: Task 1119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1119 has memoization hash 9f4cd5b2f8ab2831da5bd1af642dc674 parsl.dataflow.memoization INFO: Task 1119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0865c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1119 try 0 launched on executor multi with executor id 1120 parsl.dataflow.dflow INFO: Standard output for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/35286ee2-540f-4bc4-a855-ea8032d78b4c_isrForCrosstalkSources_1125924_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/35286ee2-540f-4bc4-a855-ea8032d78b4c_isrForCrosstalkSources_1125924_25.stderr parsl.dataflow.dflow DEBUG: Task 1120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1120 has memoization hash 240c246eecb0c4cc157cd996f9be5970 parsl.dataflow.memoization INFO: Task 1120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1120 try 0 launched on executor multi with executor id 1121 parsl.dataflow.dflow INFO: Standard output for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/553d3331-8907-447d-9579-50d6961c257d_isrForCrosstalkSources_1125897_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/553d3331-8907-447d-9579-50d6961c257d_isrForCrosstalkSources_1125897_50.stderr parsl.dataflow.dflow DEBUG: Task 1121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1121 has memoization hash 82f5fc26dbccc10c0b059dd6976ffc51 parsl.dataflow.memoization INFO: Task 1121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b084040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1121 try 0 launched on executor multi with executor id 1122 parsl.dataflow.dflow INFO: Standard output for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e7c02a22-8b2c-4d6f-a782-7d668d7e1dcb_isrForCrosstalkSources_1125916_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e7c02a22-8b2c-4d6f-a782-7d668d7e1dcb_isrForCrosstalkSources_1125916_35.stderr parsl.dataflow.dflow DEBUG: Task 1122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1122 has memoization hash 89568ccad34b286850861de06215fa94 parsl.dataflow.memoization INFO: Task 1122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0842c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1122 try 0 launched on executor multi with executor id 1123 parsl.dataflow.dflow INFO: Standard output for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/f787d9a9-b3b6-479d-a6bf-f526678fac57_isrForCrosstalkSources_1125912_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/f787d9a9-b3b6-479d-a6bf-f526678fac57_isrForCrosstalkSources_1125912_18.stderr parsl.dataflow.dflow DEBUG: Task 1123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1123 has memoization hash 32be017454d4cf87d6d4d849d6a1bda1 parsl.dataflow.memoization INFO: Task 1123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b086660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1123 try 0 launched on executor multi with executor id 1124 parsl.dataflow.dflow INFO: Standard output for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/3920175d-b8b2-4d5c-97e0-6c7badff5e94_isrForCrosstalkSources_1125900_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/3920175d-b8b2-4d5c-97e0-6c7badff5e94_isrForCrosstalkSources_1125900_4.stderr parsl.dataflow.dflow DEBUG: Task 1124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1124 has memoization hash 9094fe8906f3450f82cc66457b4466c9 parsl.dataflow.memoization INFO: Task 1124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0879c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1124 try 0 launched on executor multi with executor id 1125 parsl.dataflow.dflow INFO: Standard output for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/2ab85107-1602-40e9-b6ba-419bb72f0230_isrForCrosstalkSources_1125897_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/2ab85107-1602-40e9-b6ba-419bb72f0230_isrForCrosstalkSources_1125897_45.stderr parsl.dataflow.dflow DEBUG: Task 1125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1125 has memoization hash bd3692ca93b36c7ffa2ae112229a1415 parsl.dataflow.memoization INFO: Task 1125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:45 proc_lsst.multi:170 [INFO] [multi] job_id 16896398 proc_lsst.multi INFO: [multi] job_id 16896398 2024-03-07 15:06:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 15:06:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896398 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896398 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16896398 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-07 15:06:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-07 15:06:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 15:06:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1125 try 0 launched on executor multi with executor id 1126 parsl.dataflow.dflow INFO: Standard output for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/43ec69ee-3ab4-43f2-a9f8-ccec986069e2_isrForCrosstalkSources_1125894_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/43ec69ee-3ab4-43f2-a9f8-ccec986069e2_isrForCrosstalkSources_1125894_54.stderr parsl.dataflow.dflow DEBUG: Task 1126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1126 has memoization hash 46698fd2b3974450ef868c258e08feee parsl.dataflow.memoization INFO: Task 1126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1126 try 0 launched on executor multi with executor id 1127 parsl.dataflow.dflow INFO: Standard output for task 1126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/d30c4c02-a1f9-4f8a-be00-6a3ddf0bacfd_isrForCrosstalkSources_1125938_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/d30c4c02-a1f9-4f8a-be00-6a3ddf0bacfd_isrForCrosstalkSources_1125938_7.stderr parsl.dataflow.dflow DEBUG: Task 1127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1127 has memoization hash 84afaa4ba087607362f6d818816b11e0 parsl.dataflow.memoization INFO: Task 1127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1127 try 0 launched on executor multi with executor id 1128 parsl.dataflow.dflow INFO: Standard output for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e6f3efd6-686b-40c0-a8c0-c77c52c4a34f_isrForCrosstalkSources_1125937_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e6f3efd6-686b-40c0-a8c0-c77c52c4a34f_isrForCrosstalkSources_1125937_40.stderr parsl.dataflow.dflow DEBUG: Task 1128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1128 has memoization hash 73defe6b79f45e9f06b22202368d6c67 parsl.dataflow.memoization INFO: Task 1128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b087ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1128 try 0 launched on executor multi with executor id 1129 parsl.dataflow.dflow INFO: Standard output for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8a405fd0-e79d-4b38-996b-175f06c3efa8_isrForCrosstalkSources_1125891_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8a405fd0-e79d-4b38-996b-175f06c3efa8_isrForCrosstalkSources_1125891_1.stderr parsl.dataflow.dflow DEBUG: Task 1129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1129 has memoization hash a59410ae053250c6e2e6ca4cb75a2536 parsl.dataflow.memoization INFO: Task 1129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1129 try 0 launched on executor multi with executor id 1130 parsl.dataflow.dflow INFO: Standard output for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/69378571-32ad-4453-aa02-6363dba5a6d7_isrForCrosstalkSources_1125889_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/69378571-32ad-4453-aa02-6363dba5a6d7_isrForCrosstalkSources_1125889_9.stderr parsl.dataflow.dflow DEBUG: Task 1130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1130 has memoization hash 5e2f57f48325588b0a6798f00d4b037e parsl.dataflow.memoization INFO: Task 1130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b085a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1130 try 0 launched on executor multi with executor id 1131 parsl.dataflow.dflow INFO: Standard output for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/52470dde-6947-4075-a010-08a7044a85cf_isrForCrosstalkSources_1125917_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/52470dde-6947-4075-a010-08a7044a85cf_isrForCrosstalkSources_1125917_20.stderr parsl.dataflow.dflow DEBUG: Task 1131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1131 has memoization hash cf8919f77a1ed87b1cc69d8034d77c85 parsl.dataflow.memoization INFO: Task 1131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ec0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1131 try 0 launched on executor multi with executor id 1132 parsl.dataflow.dflow INFO: Standard output for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/e570e47c-1e79-4ed6-9fe9-fb729f5612a8_isrForCrosstalkSources_1125941_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/e570e47c-1e79-4ed6-9fe9-fb729f5612a8_isrForCrosstalkSources_1125941_49.stderr parsl.dataflow.dflow DEBUG: Task 1132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1132 has memoization hash d304c39e1c46b03bc5face8f5b6ba545 parsl.dataflow.memoization INFO: Task 1132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ef560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1132 try 0 launched on executor multi with executor id 1133 parsl.dataflow.dflow INFO: Standard output for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/9b166ae8-d90d-4ed6-bccc-727b605bb2bb_isrForCrosstalkSources_1125895_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/9b166ae8-d90d-4ed6-bccc-727b605bb2bb_isrForCrosstalkSources_1125895_12.stderr parsl.dataflow.dflow DEBUG: Task 1133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1133 has memoization hash a3cee8193a083aba56810ed71b5057f5 parsl.dataflow.memoization INFO: Task 1133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ef6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1133 try 0 launched on executor multi with executor id 1134 parsl.dataflow.dflow INFO: Standard output for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/9d34351e-f85d-4308-990c-4ee0749e112b_isrForCrosstalkSources_1125893_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/9d34351e-f85d-4308-990c-4ee0749e112b_isrForCrosstalkSources_1125893_2.stderr parsl.dataflow.dflow DEBUG: Task 1134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1134 has memoization hash b71ed33efca2e1e62a7ae5015f963821 parsl.dataflow.memoization INFO: Task 1134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ef4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1134 try 0 launched on executor multi with executor id 1135 parsl.dataflow.dflow INFO: Standard output for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f78f75b6-b7fc-4151-812d-0b1cfbd57ece_isrForCrosstalkSources_1125910_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f78f75b6-b7fc-4151-812d-0b1cfbd57ece_isrForCrosstalkSources_1125910_50.stderr parsl.dataflow.dflow DEBUG: Task 1135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1135 has memoization hash d74d19caae2435a580a92145137f3f3d parsl.dataflow.memoization INFO: Task 1135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ef420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1135 try 0 launched on executor multi with executor id 1136 parsl.dataflow.dflow INFO: Standard output for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/80119163-d0f1-4381-9ca7-f19ee8f64d6b_isrForCrosstalkSources_1125932_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/80119163-d0f1-4381-9ca7-f19ee8f64d6b_isrForCrosstalkSources_1125932_21.stderr parsl.dataflow.dflow DEBUG: Task 1136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1136 has memoization hash 2237565c097e3142d1c05a91518309bd parsl.dataflow.memoization INFO: Task 1136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ec400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1136 try 0 launched on executor multi with executor id 1137 parsl.dataflow.dflow INFO: Standard output for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/e36fe90c-c5f7-4bb5-b0b6-6fea51c1cbb9_isrForCrosstalkSources_1125891_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/e36fe90c-c5f7-4bb5-b0b6-6fea51c1cbb9_isrForCrosstalkSources_1125891_31.stderr parsl.dataflow.dflow DEBUG: Task 1137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1137 has memoization hash e6806b2c58cf66618f16de6319dc2985 parsl.dataflow.memoization INFO: Task 1137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ec540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1137 try 0 launched on executor multi with executor id 1138 parsl.dataflow.dflow INFO: Standard output for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/629246da-defe-4cae-91ef-8e847b5dc412_isrForCrosstalkSources_1125894_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/629246da-defe-4cae-91ef-8e847b5dc412_isrForCrosstalkSources_1125894_16.stderr parsl.dataflow.dflow DEBUG: Task 1138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1138 has memoization hash 2fe8167b07a858b83bff0f39c653ad67 parsl.dataflow.memoization INFO: Task 1138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ec360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1138 try 0 launched on executor multi with executor id 1139 parsl.dataflow.dflow INFO: Standard output for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/6d88f5dd-380e-42e3-97a4-4a085b0d1268_isrForCrosstalkSources_1125943_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/6d88f5dd-380e-42e3-97a4-4a085b0d1268_isrForCrosstalkSources_1125943_24.stderr parsl.dataflow.dflow DEBUG: Task 1139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1139 has memoization hash 54a2745807ce58bec78dca1acf93cbe5 parsl.dataflow.memoization INFO: Task 1139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ec5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1139 try 0 launched on executor multi with executor id 1140 parsl.dataflow.dflow INFO: Standard output for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/bfbbd503-e912-47b4-b3e7-446a739d6bca_isrForCrosstalkSources_1125893_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/bfbbd503-e912-47b4-b3e7-446a739d6bca_isrForCrosstalkSources_1125893_31.stderr parsl.dataflow.dflow DEBUG: Task 1140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1140 has memoization hash fea5c79397f7656875df90a55a93373f parsl.dataflow.memoization INFO: Task 1140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ec680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1140 try 0 launched on executor multi with executor id 1141 parsl.dataflow.dflow INFO: Standard output for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/f75baec3-7101-4c12-8e83-8c17754e28d1_isrForCrosstalkSources_1125925_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/f75baec3-7101-4c12-8e83-8c17754e28d1_isrForCrosstalkSources_1125925_21.stderr parsl.dataflow.dflow DEBUG: Task 1141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1141 has memoization hash 77c9abd323ac8b1303d95abea7079a06 parsl.dataflow.memoization INFO: Task 1141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ec4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1141 try 0 launched on executor multi with executor id 1142 parsl.dataflow.dflow INFO: Standard output for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/ee7e8d06-559f-42b6-831b-d608392053bb_isrForCrosstalkSources_1125927_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/ee7e8d06-559f-42b6-831b-d608392053bb_isrForCrosstalkSources_1125927_32.stderr parsl.dataflow.dflow DEBUG: Task 1142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1142 has memoization hash ceea56bff037ea53845320e201a476dc parsl.dataflow.memoization INFO: Task 1142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ec7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1142 try 0 launched on executor multi with executor id 1143 parsl.dataflow.dflow INFO: Standard output for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/376d5a07-6486-44de-bd0a-560d037ea2ab_isrForCrosstalkSources_1125915_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/376d5a07-6486-44de-bd0a-560d037ea2ab_isrForCrosstalkSources_1125915_48.stderr parsl.dataflow.dflow DEBUG: Task 1143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1143 has memoization hash 316daa1b6d7274d946c03d7d791ff248 parsl.dataflow.memoization INFO: Task 1143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ec720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1143 try 0 launched on executor multi with executor id 1144 parsl.dataflow.dflow INFO: Standard output for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/dc694308-5dc1-4129-97cb-9e1ac8f9202c_isrForCrosstalkSources_1125921_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/dc694308-5dc1-4129-97cb-9e1ac8f9202c_isrForCrosstalkSources_1125921_12.stderr parsl.dataflow.dflow DEBUG: Task 1144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1144 has memoization hash 891b49feb654de7bd0bda332e42e411d parsl.dataflow.memoization INFO: Task 1144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ec860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1144 try 0 launched on executor multi with executor id 1145 parsl.dataflow.dflow INFO: Standard output for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/4347d443-6918-48a3-9295-9ed6e9b1c482_isrForCrosstalkSources_1125914_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/4347d443-6918-48a3-9295-9ed6e9b1c482_isrForCrosstalkSources_1125914_12.stderr parsl.dataflow.dflow DEBUG: Task 1145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1145 has memoization hash a7fb5b1e27d4c78d7432bac5cc0cf882 parsl.dataflow.memoization INFO: Task 1145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ec9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1145 try 0 launched on executor multi with executor id 1146 parsl.dataflow.dflow INFO: Standard output for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/a7b571e8-38da-4924-bb5b-deaa0ba06065_isrForCrosstalkSources_1125899_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/a7b571e8-38da-4924-bb5b-deaa0ba06065_isrForCrosstalkSources_1125899_10.stderr parsl.dataflow.dflow DEBUG: Task 1146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1146 has memoization hash 01320f4ba11b42f78e970e1e672ddd9e parsl.dataflow.memoization INFO: Task 1146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0eca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1146 try 0 launched on executor multi with executor id 1147 parsl.dataflow.dflow INFO: Standard output for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/17a736bf-3aae-4d52-b376-067f7d36df1f_isrForCrosstalkSources_1125908_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/17a736bf-3aae-4d52-b376-067f7d36df1f_isrForCrosstalkSources_1125908_28.stderr parsl.dataflow.dflow DEBUG: Task 1147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1147 has memoization hash 0f45f81bbf2d252a848b73a2e6c8c632 parsl.dataflow.memoization INFO: Task 1147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ecae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1147 try 0 launched on executor multi with executor id 1148 parsl.dataflow.dflow INFO: Standard output for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/c85e55c5-9363-4c31-ada0-683bb31de7fe_isrForCrosstalkSources_1125901_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/c85e55c5-9363-4c31-ada0-683bb31de7fe_isrForCrosstalkSources_1125901_6.stderr parsl.dataflow.dflow DEBUG: Task 1148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1148 has memoization hash 855ce9c7c249bfe6801738d3a365ed3a parsl.dataflow.memoization INFO: Task 1148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ec900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1148 try 0 launched on executor multi with executor id 1149 parsl.dataflow.dflow INFO: Standard output for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/e372b0d6-27af-43f6-a559-fd188f7a49d1_isrForCrosstalkSources_1125894_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/e372b0d6-27af-43f6-a559-fd188f7a49d1_isrForCrosstalkSources_1125894_39.stderr parsl.dataflow.dflow DEBUG: Task 1149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1149 has memoization hash 43e2443579fb830a6c1ff0d86c1dbbb9 parsl.dataflow.memoization INFO: Task 1149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ecb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1149 try 0 launched on executor multi with executor id 1150 parsl.dataflow.dflow INFO: Standard output for task 1149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/fb75aa13-e179-4e90-8cc4-6b0cc8be54fd_isrForCrosstalkSources_1125935_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/fb75aa13-e179-4e90-8cc4-6b0cc8be54fd_isrForCrosstalkSources_1125935_55.stderr parsl.dataflow.dflow DEBUG: Task 1150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1150 has memoization hash cf4a6019eeaa81e247dc83063ff93f8b parsl.dataflow.memoization INFO: Task 1150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ecc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1150 try 0 launched on executor multi with executor id 1151 parsl.dataflow.dflow INFO: Standard output for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/95d5b46e-36b9-4a5e-85a9-8046703af6a9_isrForCrosstalkSources_1125927_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/95d5b46e-36b9-4a5e-85a9-8046703af6a9_isrForCrosstalkSources_1125927_22.stderr parsl.dataflow.dflow DEBUG: Task 1151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1151 has memoization hash 861494feda3bcafff64001573c898fd3 parsl.dataflow.memoization INFO: Task 1151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ecd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1151 try 0 launched on executor multi with executor id 1152 parsl.dataflow.dflow INFO: Standard output for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/3a6cfc8d-5706-4144-93de-4e7ed7e83f90_isrForCrosstalkSources_1125947_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/3a6cfc8d-5706-4144-93de-4e7ed7e83f90_isrForCrosstalkSources_1125947_5.stderr parsl.dataflow.dflow DEBUG: Task 1152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1152 has memoization hash dc4c75e2273dc3e3ccb69ff1865b6c2e parsl.dataflow.memoization INFO: Task 1152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ecea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:45 proc_lsst.multi:170 [INFO] [multi] job_id 16896399 proc_lsst.multi INFO: [multi] job_id 16896399 2024-03-07 15:06:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 15:06:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896399 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896399 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16896399 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-07 15:06:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-07 15:06:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 15:06:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1152 try 0 launched on executor multi with executor id 1153 parsl.dataflow.dflow INFO: Standard output for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/7d764871-be75-4cf3-b99a-5ae946c8c206_isrForCrosstalkSources_1125940_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/7d764871-be75-4cf3-b99a-5ae946c8c206_isrForCrosstalkSources_1125940_33.stderr parsl.dataflow.dflow DEBUG: Task 1153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1153 has memoization hash 37dd9d7afb94e95fe6356a434a2489c4 parsl.dataflow.memoization INFO: Task 1153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ece00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1153 try 0 launched on executor multi with executor id 1154 parsl.dataflow.dflow INFO: Standard output for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e566e0ee-ac4c-4448-95c7-b3f4c0396747_isrForCrosstalkSources_1125895_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e566e0ee-ac4c-4448-95c7-b3f4c0396747_isrForCrosstalkSources_1125895_15.stderr parsl.dataflow.dflow DEBUG: Task 1154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1154 has memoization hash 0bb79be3a58e6be036affee54692ee95 parsl.dataflow.memoization INFO: Task 1154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0eccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1154 try 0 launched on executor multi with executor id 1155 parsl.dataflow.dflow INFO: Standard output for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/a1e157f5-b397-4d40-aa4b-5c0ac4f89f99_isrForCrosstalkSources_1125939_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/a1e157f5-b397-4d40-aa4b-5c0ac4f89f99_isrForCrosstalkSources_1125939_48.stderr parsl.dataflow.dflow DEBUG: Task 1155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1155 has memoization hash 4f9c6eef0b57254e4a75fb7b35b4568c parsl.dataflow.memoization INFO: Task 1155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ecf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1155 try 0 launched on executor multi with executor id 1156 parsl.dataflow.dflow INFO: Standard output for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/d7cb91a9-efe8-4f94-95cd-ebb847100ced_isrForCrosstalkSources_1125947_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/d7cb91a9-efe8-4f94-95cd-ebb847100ced_isrForCrosstalkSources_1125947_37.stderr parsl.dataflow.dflow DEBUG: Task 1156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1156 has memoization hash 242486bfc4b97eaac5314d67411c61a0 parsl.dataflow.memoization INFO: Task 1156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ec220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1156 try 0 launched on executor multi with executor id 1157 parsl.dataflow.dflow INFO: Standard output for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/af239030-165f-4949-a007-331bdb311f4d_isrForCrosstalkSources_1125932_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/af239030-165f-4949-a007-331bdb311f4d_isrForCrosstalkSources_1125932_45.stderr parsl.dataflow.dflow DEBUG: Task 1157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1157 has memoization hash 5ea25964be7b48dbe626abdd381da58a parsl.dataflow.memoization INFO: Task 1157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0eff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1157 try 0 launched on executor multi with executor id 1158 parsl.dataflow.dflow INFO: Standard output for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/f478f5ed-9047-4d50-8c38-da2b805e3f08_isrForCrosstalkSources_1125944_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/f478f5ed-9047-4d50-8c38-da2b805e3f08_isrForCrosstalkSources_1125944_3.stderr parsl.dataflow.dflow DEBUG: Task 1158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1158 has memoization hash d82a9526d353555411d482be3805f669 parsl.dataflow.memoization INFO: Task 1158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ed080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1158 try 0 launched on executor multi with executor id 1159 parsl.dataflow.dflow INFO: Standard output for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/caf1fdbe-9372-4fae-a47c-d5e8f0abac23_isrForCrosstalkSources_1125936_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/caf1fdbe-9372-4fae-a47c-d5e8f0abac23_isrForCrosstalkSources_1125936_46.stderr parsl.dataflow.dflow DEBUG: Task 1159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1159 has memoization hash 27e148236076154a4f1579011c3d735a parsl.dataflow.memoization INFO: Task 1159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ec180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1159 try 0 launched on executor multi with executor id 1160 parsl.dataflow.dflow INFO: Standard output for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/abb955fb-cddd-445c-9973-26a9f25b46c2_isrForCrosstalkSources_1125890_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/abb955fb-cddd-445c-9973-26a9f25b46c2_isrForCrosstalkSources_1125890_2.stderr parsl.dataflow.dflow DEBUG: Task 1160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1160 has memoization hash f405452e109e7cfa78ceb13f9445f389 parsl.dataflow.memoization INFO: Task 1160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ecfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1160 try 0 launched on executor multi with executor id 1161 parsl.dataflow.dflow INFO: Standard output for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9ed0fe5c-515d-4295-9bec-b4dfd23bf501_isrForCrosstalkSources_1125934_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9ed0fe5c-515d-4295-9bec-b4dfd23bf501_isrForCrosstalkSources_1125934_42.stderr parsl.dataflow.dflow DEBUG: Task 1161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1161 has memoization hash 1564754a1d0c8dcdb2f7f179662955f4 parsl.dataflow.memoization INFO: Task 1161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0efec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1161 try 0 launched on executor multi with executor id 1162 parsl.dataflow.dflow INFO: Standard output for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d09f144f-4f87-4db5-8795-464d7bacdbf5_isrForCrosstalkSources_1125923_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d09f144f-4f87-4db5-8795-464d7bacdbf5_isrForCrosstalkSources_1125923_51.stderr parsl.dataflow.dflow DEBUG: Task 1162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1162 has memoization hash 7920fd150b1101f1a739b220ad737aa9 parsl.dataflow.memoization INFO: Task 1162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ed3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1162 try 0 launched on executor multi with executor id 1163 parsl.dataflow.dflow INFO: Standard output for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/597cc6d5-dc52-4760-8676-e04db801eebd_isrForCrosstalkSources_1125943_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/597cc6d5-dc52-4760-8676-e04db801eebd_isrForCrosstalkSources_1125943_22.stderr parsl.dataflow.dflow DEBUG: Task 1163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1163 has memoization hash 67fea63724a18a43b90133e77de35af3 parsl.dataflow.memoization INFO: Task 1163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ed440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1163 try 0 launched on executor multi with executor id 1164 parsl.dataflow.dflow INFO: Standard output for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/2848d32c-3d60-415b-a64f-2883b2579ef1_isrForCrosstalkSources_1125925_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/2848d32c-3d60-415b-a64f-2883b2579ef1_isrForCrosstalkSources_1125925_9.stderr parsl.dataflow.dflow DEBUG: Task 1164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1164 has memoization hash 2fc25c1595f2ac085ec51de5c6665ff2 parsl.dataflow.memoization INFO: Task 1164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ed4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1164 try 0 launched on executor multi with executor id 1165 parsl.dataflow.dflow INFO: Standard output for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/8f889799-b4d7-4dda-bb45-a5231f455ae0_isrForCrosstalkSources_1125895_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/8f889799-b4d7-4dda-bb45-a5231f455ae0_isrForCrosstalkSources_1125895_4.stderr parsl.dataflow.dflow DEBUG: Task 1165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1165 has memoization hash c7d74e0ff59ac158e231501f0d13b4c9 parsl.dataflow.memoization INFO: Task 1165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ed260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1165 try 0 launched on executor multi with executor id 1166 parsl.dataflow.dflow INFO: Standard output for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/2e4232ee-70b3-4216-ac1d-82edb063591a_isrForCrosstalkSources_1125908_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/2e4232ee-70b3-4216-ac1d-82edb063591a_isrForCrosstalkSources_1125908_10.stderr parsl.dataflow.dflow DEBUG: Task 1166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1166 has memoization hash ca690a2d4428a75bb407355eed97c4d4 parsl.dataflow.memoization INFO: Task 1166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ed620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1166 try 0 launched on executor multi with executor id 1167 parsl.dataflow.dflow INFO: Standard output for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2e3a2f2f-65d8-4c10-9c99-230e4e56094b_isrForCrosstalkSources_1125895_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2e3a2f2f-65d8-4c10-9c99-230e4e56094b_isrForCrosstalkSources_1125895_24.stderr parsl.dataflow.dflow DEBUG: Task 1167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1167 has memoization hash 57c4c8b9788d6b9ee8ec827b52e5b545 parsl.dataflow.memoization INFO: Task 1167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ed580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1167 try 0 launched on executor multi with executor id 1168 parsl.dataflow.dflow INFO: Standard output for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/84039217-2f19-4512-bb6a-0c9e9853812b_isrForCrosstalkSources_1125939_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/84039217-2f19-4512-bb6a-0c9e9853812b_isrForCrosstalkSources_1125939_7.stderr parsl.dataflow.dflow DEBUG: Task 1168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1168 has memoization hash bd5bcba7c8f6f03921f172b538553f7e parsl.dataflow.memoization INFO: Task 1168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ed760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1168 try 0 launched on executor multi with executor id 1169 parsl.dataflow.dflow INFO: Standard output for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/c2766d34-c785-4807-84ac-59beb6612976_isrForCrosstalkSources_1125923_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/c2766d34-c785-4807-84ac-59beb6612976_isrForCrosstalkSources_1125923_52.stderr parsl.dataflow.dflow DEBUG: Task 1169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1169 has memoization hash 69b58b94d90457ab2fa4a4df697911b3 parsl.dataflow.memoization INFO: Task 1169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ed6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1169 try 0 launched on executor multi with executor id 1170 parsl.dataflow.dflow INFO: Standard output for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/a037c7b3-d473-4177-bc25-489fa9aade01_isrForCrosstalkSources_1125896_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/a037c7b3-d473-4177-bc25-489fa9aade01_isrForCrosstalkSources_1125896_21.stderr parsl.dataflow.dflow DEBUG: Task 1170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1170 has memoization hash d43b61deae843e77c674997bf9d839fb parsl.dataflow.memoization INFO: Task 1170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ed940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1170 try 0 launched on executor multi with executor id 1171 parsl.dataflow.dflow INFO: Standard output for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/84744ce2-72fd-464b-82c1-64ca60c0776a_isrForCrosstalkSources_1125909_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/84744ce2-72fd-464b-82c1-64ca60c0776a_isrForCrosstalkSources_1125909_47.stderr parsl.dataflow.dflow DEBUG: Task 1171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1171 has memoization hash 0b1ffc56a9efe140b6185b2b5a5a32bd parsl.dataflow.memoization INFO: Task 1171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ed800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1171 try 0 launched on executor multi with executor id 1172 parsl.dataflow.dflow INFO: Standard output for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/4a59661e-9aed-4fed-809e-0142e7acea6a_isrForCrosstalkSources_1125943_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/4a59661e-9aed-4fed-809e-0142e7acea6a_isrForCrosstalkSources_1125943_52.stderr parsl.dataflow.dflow DEBUG: Task 1172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1172 has memoization hash de871df7fb067199ce54a98f3a649afa parsl.dataflow.memoization INFO: Task 1172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ed8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1172 try 0 launched on executor multi with executor id 1173 parsl.dataflow.dflow INFO: Standard output for task 1172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/6ebe943a-9015-4801-b746-581445cb89c0_isrForCrosstalkSources_1125905_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/6ebe943a-9015-4801-b746-581445cb89c0_isrForCrosstalkSources_1125905_60.stderr parsl.dataflow.dflow DEBUG: Task 1173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1173 has memoization hash e561dfb9eb3c13131f2a422627a55346 parsl.dataflow.memoization INFO: Task 1173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0edb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1173 try 0 launched on executor multi with executor id 1174 parsl.dataflow.dflow INFO: Standard output for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/e7246c9b-deff-4b77-ad84-60107b6d6df4_isrForCrosstalkSources_1125919_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/e7246c9b-deff-4b77-ad84-60107b6d6df4_isrForCrosstalkSources_1125919_33.stderr parsl.dataflow.dflow DEBUG: Task 1174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1174 has memoization hash eed8d30e388775d6751d72681b1d96dc parsl.dataflow.memoization INFO: Task 1174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0eda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1174 try 0 launched on executor multi with executor id 1175 parsl.dataflow.dflow INFO: Standard output for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/0e390301-90f8-4fe5-977c-9ec960a30abc_isrForCrosstalkSources_1125897_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/0e390301-90f8-4fe5-977c-9ec960a30abc_isrForCrosstalkSources_1125897_3.stderr parsl.dataflow.dflow DEBUG: Task 1175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1175 submitted for App isrForCrosstalkSources, not waiting on any dependency 2024-03-07 15:06:46 proc_lsst.multi:170 [INFO] [multi] job_id 16896400 proc_lsst.multi INFO: [multi] job_id 16896400 2024-03-07 15:06:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 15:06:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896400 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896400 parsl.dataflow.dflow DEBUG: Task 1175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1175 has memoization hash 0228719dd3ea4930f111ad91f21e5f96 parsl.dataflow.memoization INFO: Task 1175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0edc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16896400 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-07 15:06:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-07 15:06:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 15:06:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1175 try 0 launched on executor multi with executor id 1176 parsl.dataflow.dflow INFO: Standard output for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/23273230-27ed-4631-882d-e6fc17daf858_isrForCrosstalkSources_1125931_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/23273230-27ed-4631-882d-e6fc17daf858_isrForCrosstalkSources_1125931_19.stderr parsl.dataflow.dflow DEBUG: Task 1176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1176 has memoization hash 9fe8dce10349ec267e9ef3ff0cc908ae parsl.dataflow.memoization INFO: Task 1176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1176 try 0 launched on executor multi with executor id 1177 parsl.dataflow.dflow INFO: Standard output for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/69032172-33ca-43a6-8b22-0650d1a46ca5_isrForCrosstalkSources_1125923_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/69032172-33ca-43a6-8b22-0650d1a46ca5_isrForCrosstalkSources_1125923_32.stderr parsl.dataflow.dflow DEBUG: Task 1177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1177 has memoization hash 83948121f3d34950e567120fcf871077 parsl.dataflow.memoization INFO: Task 1177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ec2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1177 try 0 launched on executor multi with executor id 1178 parsl.dataflow.dflow INFO: Standard output for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/746a7d7b-f23a-4c15-aac7-f09771376933_isrForCrosstalkSources_1125908_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/746a7d7b-f23a-4c15-aac7-f09771376933_isrForCrosstalkSources_1125908_56.stderr parsl.dataflow.dflow DEBUG: Task 1178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1178 has memoization hash 267d8304a485d936f2e86124bc13f90d parsl.dataflow.memoization INFO: Task 1178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1178 try 0 launched on executor multi with executor id 1179 parsl.dataflow.dflow INFO: Standard output for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/3cb4f70f-7b88-42b2-b5d4-80bfed94d45b_isrForCrosstalkSources_1125898_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/3cb4f70f-7b88-42b2-b5d4-80bfed94d45b_isrForCrosstalkSources_1125898_12.stderr parsl.dataflow.dflow DEBUG: Task 1179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1179 has memoization hash 1ddb008ef3dbe8be09abdfd8569a2e3d parsl.dataflow.memoization INFO: Task 1179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0edda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1179 try 0 launched on executor multi with executor id 1180 parsl.dataflow.dflow INFO: Standard output for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/92017e0f-7d4f-4c56-a721-bcb656324728_isrForCrosstalkSources_1125935_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/92017e0f-7d4f-4c56-a721-bcb656324728_isrForCrosstalkSources_1125935_52.stderr parsl.dataflow.dflow DEBUG: Task 1180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1180 has memoization hash ff5322b7de89ad82d2db85c778676db7 parsl.dataflow.memoization INFO: Task 1180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ed1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1180 try 0 launched on executor multi with executor id 1181 parsl.dataflow.dflow INFO: Standard output for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/d0a6cf8d-698a-41a3-a325-5b59002ddcb4_isrForCrosstalkSources_1125938_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/d0a6cf8d-698a-41a3-a325-5b59002ddcb4_isrForCrosstalkSources_1125938_52.stderr parsl.dataflow.dflow DEBUG: Task 1181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1181 has memoization hash bef3385975b1fd41daaf7a7f4c21def8 parsl.dataflow.memoization INFO: Task 1181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1181 try 0 launched on executor multi with executor id 1182 parsl.dataflow.dflow INFO: Standard output for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/4c9dbf7b-4baf-452a-8f1f-3aaebd35609e_isrForCrosstalkSources_1125923_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/4c9dbf7b-4baf-452a-8f1f-3aaebd35609e_isrForCrosstalkSources_1125923_4.stderr parsl.dataflow.dflow DEBUG: Task 1182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1182 has memoization hash 85e3a9ab5e42fdcd8a6548397e758cfa parsl.dataflow.memoization INFO: Task 1182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1182 try 0 launched on executor multi with executor id 1183 parsl.dataflow.dflow INFO: Standard output for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/d6f9a53f-7abd-48c2-b808-211aa86fa3c3_isrForCrosstalkSources_1125901_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/d6f9a53f-7abd-48c2-b808-211aa86fa3c3_isrForCrosstalkSources_1125901_48.stderr parsl.dataflow.dflow DEBUG: Task 1183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1183 has memoization hash 3ebf8b8c5a6cad6947d1f8c5a0727dea parsl.dataflow.memoization INFO: Task 1183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1183 try 0 launched on executor multi with executor id 1184 parsl.dataflow.dflow INFO: Standard output for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/635c9963-df1e-452e-9f3a-c15475c16082_isrForCrosstalkSources_1125916_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/635c9963-df1e-452e-9f3a-c15475c16082_isrForCrosstalkSources_1125916_9.stderr parsl.dataflow.dflow DEBUG: Task 1184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1184 has memoization hash 92b0dbe2033464bda48be93b5f24403f parsl.dataflow.memoization INFO: Task 1184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ee020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1184 try 0 launched on executor multi with executor id 1185 parsl.dataflow.dflow INFO: Standard output for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/b380be05-ba8c-4bf7-8b76-f6be4d430a27_isrForCrosstalkSources_1125909_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/b380be05-ba8c-4bf7-8b76-f6be4d430a27_isrForCrosstalkSources_1125909_13.stderr parsl.dataflow.dflow DEBUG: Task 1185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1185 has memoization hash 4eb2b0d604f0c558f28ab5aebbb2f380 parsl.dataflow.memoization INFO: Task 1185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0edee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1185 try 0 launched on executor multi with executor id 1186 parsl.dataflow.dflow INFO: Standard output for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d627eb0a-ecb8-4f73-90dd-ba11e183ee3f_isrForCrosstalkSources_1125919_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d627eb0a-ecb8-4f73-90dd-ba11e183ee3f_isrForCrosstalkSources_1125919_21.stderr parsl.dataflow.dflow DEBUG: Task 1186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1186 has memoization hash f784e82516e09ac66e8c0ebc558cc3f5 parsl.dataflow.memoization INFO: Task 1186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ee0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1186 try 0 launched on executor multi with executor id 1187 parsl.dataflow.dflow INFO: Standard output for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/969c9918-9a44-4d30-9f7e-35177d9e2066_isrForCrosstalkSources_1125906_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/969c9918-9a44-4d30-9f7e-35177d9e2066_isrForCrosstalkSources_1125906_60.stderr parsl.dataflow.dflow DEBUG: Task 1187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1187 has memoization hash aa5008c2586d5d12bafd7767e3c11614 parsl.dataflow.memoization INFO: Task 1187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ede40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1187 try 0 launched on executor multi with executor id 1188 parsl.dataflow.dflow INFO: Standard output for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a22f7ed7-2864-4a00-b969-1fa5a58519e7_isrForCrosstalkSources_1125938_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a22f7ed7-2864-4a00-b969-1fa5a58519e7_isrForCrosstalkSources_1125938_35.stderr parsl.dataflow.dflow DEBUG: Task 1188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1188 has memoization hash c607ebd16b61361af1c95581fee0c342 parsl.dataflow.memoization INFO: Task 1188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ee160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1188 try 0 launched on executor multi with executor id 1189 parsl.dataflow.dflow INFO: Standard output for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/e0a61e85-52d5-45f4-8882-7beee6355a3a_isrForCrosstalkSources_1125913_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/e0a61e85-52d5-45f4-8882-7beee6355a3a_isrForCrosstalkSources_1125913_26.stderr parsl.dataflow.dflow DEBUG: Task 1189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1189 has memoization hash 1e79f0fe3362e96d0c3b68d463f9c028 parsl.dataflow.memoization INFO: Task 1189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ee2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1189 try 0 launched on executor multi with executor id 1190 parsl.dataflow.dflow INFO: Standard output for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/96db6dfc-0593-4c9e-a14c-540116c3e4fe_isrForCrosstalkSources_1125901_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/96db6dfc-0593-4c9e-a14c-540116c3e4fe_isrForCrosstalkSources_1125901_34.stderr parsl.dataflow.dflow DEBUG: Task 1190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1190 has memoization hash fcf14cf1fe976aacbbfd364a16717476 parsl.dataflow.memoization INFO: Task 1190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ef2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1190 try 0 launched on executor multi with executor id 1191 parsl.dataflow.dflow INFO: Standard output for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/866cab21-8d68-48e6-a2d9-7e5c12f262c5_isrForCrosstalkSources_1125899_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/866cab21-8d68-48e6-a2d9-7e5c12f262c5_isrForCrosstalkSources_1125899_45.stderr parsl.dataflow.dflow DEBUG: Task 1191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1191 has memoization hash 9f913efd471917dfb9dfd4affa36b1e3 parsl.dataflow.memoization INFO: Task 1191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ef380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1191 try 0 launched on executor multi with executor id 1192 parsl.dataflow.dflow INFO: Standard output for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/64c15eb7-7ab9-42e7-acd2-605d72c472c6_isrForCrosstalkSources_1125923_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/64c15eb7-7ab9-42e7-acd2-605d72c472c6_isrForCrosstalkSources_1125923_16.stderr parsl.dataflow.dflow DEBUG: Task 1192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1192 has memoization hash 7bead73593eaee5fcdd2e071ab470fc6 parsl.dataflow.memoization INFO: Task 1192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ef100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1192 try 0 launched on executor multi with executor id 1193 parsl.dataflow.dflow INFO: Standard output for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/7b9874f5-162d-4bef-bd34-218c3ae10a7f_isrForCrosstalkSources_1125946_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/7b9874f5-162d-4bef-bd34-218c3ae10a7f_isrForCrosstalkSources_1125946_11.stderr parsl.dataflow.dflow DEBUG: Task 1193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1193 has memoization hash 8f70270ac3625c8967cb824a4f82658b parsl.dataflow.memoization INFO: Task 1193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ef240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1193 try 0 launched on executor multi with executor id 1194 parsl.dataflow.dflow INFO: Standard output for task 1193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/02c024c9-f192-4c85-aa27-527ad6ac65ed_isrForCrosstalkSources_1125922_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/02c024c9-f192-4c85-aa27-527ad6ac65ed_isrForCrosstalkSources_1125922_16.stderr parsl.dataflow.dflow DEBUG: Task 1194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1194 has memoization hash b98120856494eb1b80dc556c7f474e06 parsl.dataflow.memoization INFO: Task 1194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ef1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1194 try 0 launched on executor multi with executor id 1195 parsl.dataflow.dflow INFO: Standard output for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/4b1f0ba1-81df-46d4-86ea-2e35373df36a_isrForCrosstalkSources_1125924_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/4b1f0ba1-81df-46d4-86ea-2e35373df36a_isrForCrosstalkSources_1125924_33.stderr parsl.dataflow.dflow DEBUG: Task 1195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1195 has memoization hash f63a072b192dff6c4e9db4c666e21798 parsl.dataflow.memoization INFO: Task 1195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0eefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1195 try 0 launched on executor multi with executor id 1196 parsl.dataflow.dflow INFO: Standard output for task 1195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/6a679216-6d4c-465c-84a3-a9d377373723_isrForCrosstalkSources_1125900_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/6a679216-6d4c-465c-84a3-a9d377373723_isrForCrosstalkSources_1125900_59.stderr parsl.dataflow.dflow DEBUG: Task 1196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1196 has memoization hash a776e27d7b8bb565473b8e7dc80be847 parsl.dataflow.memoization INFO: Task 1196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ef060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1196 try 0 launched on executor multi with executor id 1197 parsl.dataflow.dflow INFO: Standard output for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/a65455f7-b887-4543-bf08-e8c32b3fffdc_isrForCrosstalkSources_1125905_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/a65455f7-b887-4543-bf08-e8c32b3fffdc_isrForCrosstalkSources_1125905_9.stderr parsl.dataflow.dflow DEBUG: Task 1197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1197 has memoization hash cac8509ce70d66f5ba6a31d62194c145 parsl.dataflow.memoization INFO: Task 1197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0eed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1197 try 0 launched on executor multi with executor id 1198 parsl.dataflow.dflow INFO: Standard output for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/44dc33a6-1119-435c-afc7-c95502ee2f2e_isrForCrosstalkSources_1125931_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/44dc33a6-1119-435c-afc7-c95502ee2f2e_isrForCrosstalkSources_1125931_46.stderr parsl.dataflow.dflow DEBUG: Task 1198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1198 has memoization hash 3eaccee0fdb811de579789530637eaa6 parsl.dataflow.memoization INFO: Task 1198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0eee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1198 try 0 launched on executor multi with executor id 1199 parsl.dataflow.dflow INFO: Standard output for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/10ac8086-7eca-4b69-bad4-14fbb5250ed1_isrForCrosstalkSources_1125907_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/10ac8086-7eca-4b69-bad4-14fbb5250ed1_isrForCrosstalkSources_1125907_5.stderr parsl.dataflow.dflow DEBUG: Task 1199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1199 has memoization hash c1eef54b7a0c0b4cee7de145f0789243 parsl.dataflow.memoization INFO: Task 1199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0eeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1199 try 0 launched on executor multi with executor id 1200 parsl.dataflow.dflow INFO: Standard output for task 1199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/17a7d32e-65c0-4892-a591-6d143e3b458c_isrForCrosstalkSources_1125934_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/17a7d32e-65c0-4892-a591-6d143e3b458c_isrForCrosstalkSources_1125934_37.stderr parsl.dataflow.dflow DEBUG: Task 1200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1200 has memoization hash ad9b1806188eef4c837856bc28a79e44 parsl.dataflow.memoization INFO: Task 1200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0eede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1200 try 0 launched on executor multi with executor id 1201 parsl.dataflow.dflow INFO: Standard output for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/90cf62bf-1145-40cf-9e35-578c928fa387_isrForCrosstalkSources_1125942_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/90cf62bf-1145-40cf-9e35-578c928fa387_isrForCrosstalkSources_1125942_12.stderr parsl.dataflow.dflow DEBUG: Task 1201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1201 has memoization hash 340c0b97d0f40d9ab62cb337611312ca parsl.dataflow.memoization INFO: Task 1201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0eef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1201 try 0 launched on executor multi with executor id 1202 parsl.dataflow.dflow INFO: Standard output for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/1325b789-00a3-4827-a58a-2133227148e9_isrForCrosstalkSources_1125898_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/1325b789-00a3-4827-a58a-2133227148e9_isrForCrosstalkSources_1125898_18.stderr parsl.dataflow.dflow DEBUG: Task 1202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1202 has memoization hash 0b3e78028aad8e54f85188623643c9da parsl.dataflow.memoization INFO: Task 1202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0eea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1202 try 0 launched on executor multi with executor id 1203 parsl.dataflow.dflow INFO: Standard output for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/3d760f5f-484b-48b8-8312-9f84ffa2c459_isrForCrosstalkSources_1125892_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/3d760f5f-484b-48b8-8312-9f84ffa2c459_isrForCrosstalkSources_1125892_39.stderr parsl.dataflow.dflow DEBUG: Task 1203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1203 has memoization hash 23070d2df6230d4923627d86c48edf60 parsl.dataflow.memoization INFO: Task 1203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0eeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1203 try 0 launched on executor multi with executor id 1204 parsl.dataflow.dflow INFO: Standard output for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/17e38d45-3679-4060-8935-3127d9f38724_isrForCrosstalkSources_1125913_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/17e38d45-3679-4060-8935-3127d9f38724_isrForCrosstalkSources_1125913_47.stderr parsl.dataflow.dflow DEBUG: Task 1204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1204 has memoization hash 042ca2d5606abd6bfafd3856f8cb25bc parsl.dataflow.memoization INFO: Task 1204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0eec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:46 proc_lsst.multi:170 [INFO] [multi] job_id 16896401 proc_lsst.multi INFO: [multi] job_id 16896401 2024-03-07 15:06:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 15:06:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896401 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896401 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16896401 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-07 15:06:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-07 15:06:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 15:06:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1204 try 0 launched on executor multi with executor id 1205 parsl.dataflow.dflow INFO: Standard output for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/cbfa505c-0dd4-4087-9137-7f08f52045c2_isrForCrosstalkSources_1125897_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/cbfa505c-0dd4-4087-9137-7f08f52045c2_isrForCrosstalkSources_1125897_22.stderr parsl.dataflow.dflow DEBUG: Task 1205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1205 has memoization hash 302da1d4898426b49eea00e33f8daef4 parsl.dataflow.memoization INFO: Task 1205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ee8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1205 try 0 launched on executor multi with executor id 1206 parsl.dataflow.dflow INFO: Standard output for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f168732f-6c1f-41fb-8f1c-0c336be3fa1c_isrForCrosstalkSources_1125930_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f168732f-6c1f-41fb-8f1c-0c336be3fa1c_isrForCrosstalkSources_1125930_22.stderr parsl.dataflow.dflow DEBUG: Task 1206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1206 has memoization hash 4154049c2efa68df4f56bdc0b2a08897 parsl.dataflow.memoization INFO: Task 1206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0eeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1206 try 0 launched on executor multi with executor id 1207 parsl.dataflow.dflow INFO: Standard output for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/7982fa75-57c0-4d70-86d1-3c3a639b1e93_isrForCrosstalkSources_1125910_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/7982fa75-57c0-4d70-86d1-3c3a639b1e93_isrForCrosstalkSources_1125910_5.stderr parsl.dataflow.dflow DEBUG: Task 1207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1207 has memoization hash 38fa3683684cd481e7ca3edc6c2f1d1d parsl.dataflow.memoization INFO: Task 1207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0edf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1207 try 0 launched on executor multi with executor id 1208 parsl.dataflow.dflow INFO: Standard output for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/e2476425-1cfa-4b9e-9034-f925a5c26d07_isrForCrosstalkSources_1125924_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/e2476425-1cfa-4b9e-9034-f925a5c26d07_isrForCrosstalkSources_1125924_23.stderr parsl.dataflow.dflow DEBUG: Task 1208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1208 has memoization hash 3827029ac7c36f43637ee6deab475b38 parsl.dataflow.memoization INFO: Task 1208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ed9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1208 try 0 launched on executor multi with executor id 1209 parsl.dataflow.dflow INFO: Standard output for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/28aeaad6-0c71-4a08-aadb-9277eee579ea_isrForCrosstalkSources_1125933_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/28aeaad6-0c71-4a08-aadb-9277eee579ea_isrForCrosstalkSources_1125933_38.stderr parsl.dataflow.dflow DEBUG: Task 1209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1209 has memoization hash e015e2d69c9a543d57dd6baa0f272b9a parsl.dataflow.memoization INFO: Task 1209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1209 try 0 launched on executor multi with executor id 1210 parsl.dataflow.dflow INFO: Standard output for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/049160fd-eca5-42cd-b289-b89e542253cc_isrForCrosstalkSources_1125926_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/049160fd-eca5-42cd-b289-b89e542253cc_isrForCrosstalkSources_1125926_46.stderr parsl.dataflow.dflow DEBUG: Task 1210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1210 has memoization hash a877454493ae52a49243219bda1c9175 parsl.dataflow.memoization INFO: Task 1210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ee840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1210 try 0 launched on executor multi with executor id 1211 parsl.dataflow.dflow INFO: Standard output for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/83a03753-f3a8-47d6-bc61-0065edadcb63_isrForCrosstalkSources_1125944_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/83a03753-f3a8-47d6-bc61-0065edadcb63_isrForCrosstalkSources_1125944_44.stderr parsl.dataflow.dflow DEBUG: Task 1211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1211 has memoization hash be531ec42e668b4d790e3df6e31d2515 parsl.dataflow.memoization INFO: Task 1211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ee700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1211 try 0 launched on executor multi with executor id 1212 parsl.dataflow.dflow INFO: Standard output for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/ab1a3914-3abe-489c-92cd-25894f27052d_isrForCrosstalkSources_1125930_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/ab1a3914-3abe-489c-92cd-25894f27052d_isrForCrosstalkSources_1125930_49.stderr parsl.dataflow.dflow DEBUG: Task 1212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1212 has memoization hash 7f8727559eb1a81741e429469734749f parsl.dataflow.memoization INFO: Task 1212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ef600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1212 try 0 launched on executor multi with executor id 1213 parsl.dataflow.dflow INFO: Standard output for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0b8bc14d-6be8-4d98-964c-774743515d75_isrForCrosstalkSources_1125914_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0b8bc14d-6be8-4d98-964c-774743515d75_isrForCrosstalkSources_1125914_14.stderr parsl.dataflow.dflow DEBUG: Task 1213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1213 has memoization hash 7256c9c5ee58b7c78cee9dfa5b818eb6 parsl.dataflow.memoization INFO: Task 1213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ee660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1213 try 0 launched on executor multi with executor id 1214 parsl.dataflow.dflow INFO: Standard output for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/d70273ee-ccb1-4d6f-a53a-632f9747fe87_isrForCrosstalkSources_1125935_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/d70273ee-ccb1-4d6f-a53a-632f9747fe87_isrForCrosstalkSources_1125935_31.stderr parsl.dataflow.dflow DEBUG: Task 1214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1214 has memoization hash 2033fb822561bfed406515e42c117e79 parsl.dataflow.memoization INFO: Task 1214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ee520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1214 try 0 launched on executor multi with executor id 1215 parsl.dataflow.dflow INFO: Standard output for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/5ff2d7d5-cf66-4689-8db3-d44f6f6455d5_isrForCrosstalkSources_1125900_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/5ff2d7d5-cf66-4689-8db3-d44f6f6455d5_isrForCrosstalkSources_1125900_22.stderr parsl.dataflow.dflow DEBUG: Task 1215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1215 has memoization hash 2e0578934f3fd50d8a0fd8fc874fc2f7 parsl.dataflow.memoization INFO: Task 1215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0efe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1215 try 0 launched on executor multi with executor id 1216 parsl.dataflow.dflow INFO: Standard output for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/80437f5a-51e8-4ebd-b1d8-baa3f2a6b433_isrForCrosstalkSources_1125889_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/80437f5a-51e8-4ebd-b1d8-baa3f2a6b433_isrForCrosstalkSources_1125889_2.stderr parsl.dataflow.dflow DEBUG: Task 1216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1216 has memoization hash d8603cc14eb708963f31cd2e3f9fbb9c parsl.dataflow.memoization INFO: Task 1216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0efce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1216 try 0 launched on executor multi with executor id 1217 parsl.dataflow.dflow INFO: Standard output for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/7406cf65-e0f6-455d-baae-1fd2f988bf37_isrForCrosstalkSources_1125889_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/7406cf65-e0f6-455d-baae-1fd2f988bf37_isrForCrosstalkSources_1125889_1.stderr parsl.dataflow.dflow DEBUG: Task 1217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1217 has memoization hash 814fe8cc0f874ec3bf7e9cacdba3230a parsl.dataflow.memoization INFO: Task 1217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0efc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1217 try 0 launched on executor multi with executor id 1218 parsl.dataflow.dflow INFO: Standard output for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/1ecf97f4-1b70-4982-bada-6aefb1531e26_isrForCrosstalkSources_1125896_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/1ecf97f4-1b70-4982-bada-6aefb1531e26_isrForCrosstalkSources_1125896_10.stderr parsl.dataflow.dflow DEBUG: Task 1218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1218 has memoization hash 4a22c6eadf247006b53c9492a2d503d1 parsl.dataflow.memoization INFO: Task 1218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0efd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1218 try 0 launched on executor multi with executor id 1219 parsl.dataflow.dflow INFO: Standard output for task 1218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/c1e3128c-3127-4c4b-9be2-2da8dbd7a7b7_isrForCrosstalkSources_1125899_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/c1e3128c-3127-4c4b-9be2-2da8dbd7a7b7_isrForCrosstalkSources_1125899_2.stderr parsl.dataflow.dflow DEBUG: Task 1219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1219 has memoization hash 69cd6c249ed8c2b9afc1d343dae7e345 parsl.dataflow.memoization INFO: Task 1219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0efa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1219 try 0 launched on executor multi with executor id 1220 parsl.dataflow.dflow INFO: Standard output for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/90ca9050-8730-4486-b051-c790003365ea_isrForCrosstalkSources_1125917_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/90ca9050-8730-4486-b051-c790003365ea_isrForCrosstalkSources_1125917_39.stderr parsl.dataflow.dflow DEBUG: Task 1220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1220 has memoization hash 6af6f66425737ad471fb75862e7ae7b3 parsl.dataflow.memoization INFO: Task 1220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0efba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1220 try 0 launched on executor multi with executor id 1221 parsl.dataflow.dflow INFO: Standard output for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/266c5e34-15ff-4f49-9627-8044efa54ff3_isrForCrosstalkSources_1125942_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/266c5e34-15ff-4f49-9627-8044efa54ff3_isrForCrosstalkSources_1125942_32.stderr parsl.dataflow.dflow DEBUG: Task 1221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1221 has memoization hash ab795bc63d90aa05efa4cad1e1aed287 parsl.dataflow.memoization INFO: Task 1221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ef9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1221 try 0 launched on executor multi with executor id 1222 parsl.dataflow.dflow INFO: Standard output for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/5cca5be0-e49e-493e-a783-78f44cf1803d_isrForCrosstalkSources_1125903_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/5cca5be0-e49e-493e-a783-78f44cf1803d_isrForCrosstalkSources_1125903_55.stderr parsl.dataflow.dflow DEBUG: Task 1222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1222 has memoization hash c9bec00d5e8e6f2964e3eac1fc27f7fc parsl.dataflow.memoization INFO: Task 1222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ef880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1222 try 0 launched on executor multi with executor id 1223 parsl.dataflow.dflow INFO: Standard output for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/f7832ebc-6791-4730-b321-5427e65a7088_isrForCrosstalkSources_1125900_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/f7832ebc-6791-4730-b321-5427e65a7088_isrForCrosstalkSources_1125900_24.stderr parsl.dataflow.dflow DEBUG: Task 1223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1223 has memoization hash c18950b4805a3497e662276aae671cac parsl.dataflow.memoization INFO: Task 1223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0efb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1223 try 0 launched on executor multi with executor id 1224 parsl.dataflow.dflow INFO: Standard output for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/8e883939-750c-45ec-a781-b0f03eb3bac3_isrForCrosstalkSources_1125912_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/8e883939-750c-45ec-a781-b0f03eb3bac3_isrForCrosstalkSources_1125912_39.stderr parsl.dataflow.dflow DEBUG: Task 1224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1224 has memoization hash ffb99fc84382666c91a12ba5bdeffd8f parsl.dataflow.memoization INFO: Task 1224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ef740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1224 try 0 launched on executor multi with executor id 1225 parsl.dataflow.dflow INFO: Standard output for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/32ef0c89-4387-4648-97e5-938f6ab397f8_isrForCrosstalkSources_1125924_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/32ef0c89-4387-4648-97e5-938f6ab397f8_isrForCrosstalkSources_1125924_37.stderr parsl.dataflow.dflow DEBUG: Task 1225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1225 has memoization hash 6200ba5ebd9205883826bd3f6bcdae22 parsl.dataflow.memoization INFO: Task 1225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ef920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1225 try 0 launched on executor multi with executor id 1226 parsl.dataflow.dflow INFO: Standard output for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/1fea3c3c-a2bb-43ee-bfa2-c140dc43aad1_isrForCrosstalkSources_1125892_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/1fea3c3c-a2bb-43ee-bfa2-c140dc43aad1_isrForCrosstalkSources_1125892_26.stderr parsl.dataflow.dflow DEBUG: Task 1226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1226 has memoization hash 445175aefe2a6366861ba92e84a5ec73 parsl.dataflow.memoization INFO: Task 1226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ee200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1226 try 0 launched on executor multi with executor id 1227 parsl.dataflow.dflow INFO: Standard output for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/0be87a6e-d4b5-4ed1-9ed0-a386b8f9b3c0_isrForCrosstalkSources_1125905_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/0be87a6e-d4b5-4ed1-9ed0-a386b8f9b3c0_isrForCrosstalkSources_1125905_53.stderr parsl.dataflow.dflow DEBUG: Task 1227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1227 has memoization hash 3843d43206fe69a8108316bc73a87d7f parsl.dataflow.memoization INFO: Task 1227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ee340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1227 try 0 launched on executor multi with executor id 1228 parsl.dataflow.dflow INFO: Standard output for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/59bae909-03c1-4d56-987c-6e1ebc68ff85_isrForCrosstalkSources_1125938_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/59bae909-03c1-4d56-987c-6e1ebc68ff85_isrForCrosstalkSources_1125938_54.stderr parsl.dataflow.dflow DEBUG: Task 1228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1228 has memoization hash 156b69576449aa94b72777426cbaba52 parsl.dataflow.memoization INFO: Task 1228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ef7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1228 try 0 launched on executor multi with executor id 1229 parsl.dataflow.dflow INFO: Standard output for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/506568c6-ac2b-419a-aad5-a423f54e7d6b_isrForCrosstalkSources_1125927_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/506568c6-ac2b-419a-aad5-a423f54e7d6b_isrForCrosstalkSources_1125927_57.stderr parsl.dataflow.dflow DEBUG: Task 1229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1229 has memoization hash 462f92d2cf6a34c830cf6541c0283219 parsl.dataflow.memoization INFO: Task 1229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ec040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1229 try 0 launched on executor multi with executor id 1230 parsl.dataflow.dflow INFO: Standard output for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/9e3a3f55-01d4-4911-9150-b09c245841a1_isrForCrosstalkSources_1125920_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/9e3a3f55-01d4-4911-9150-b09c245841a1_isrForCrosstalkSources_1125920_44.stderr parsl.dataflow.dflow DEBUG: Task 1230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1230 has memoization hash 7ad24ed7336b43f5216457ff62cc15d0 parsl.dataflow.memoization INFO: Task 1230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ee480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1230 try 0 launched on executor multi with executor id 1231 parsl.dataflow.dflow INFO: Standard output for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/5d02f9eb-c75b-4329-a745-caa690a2b0f1_isrForCrosstalkSources_1125932_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/5d02f9eb-c75b-4329-a745-caa690a2b0f1_isrForCrosstalkSources_1125932_36.stderr parsl.dataflow.dflow DEBUG: Task 1231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1231 has memoization hash 856931630f5e854bfcbe207bc0b7cf1d parsl.dataflow.memoization INFO: Task 1231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ee5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1231 try 0 launched on executor multi with executor id 1232 parsl.dataflow.dflow INFO: Standard output for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/e5803787-90ee-4da2-be94-d5d6218c1d0f_isrForCrosstalkSources_1125934_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/e5803787-90ee-4da2-be94-d5d6218c1d0f_isrForCrosstalkSources_1125934_15.stderr parsl.dataflow.dflow DEBUG: Task 1232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1232 has memoization hash ae328fbd4f20f826931ecbc2ae7165de parsl.dataflow.memoization INFO: Task 1232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ee980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:47 proc_lsst.multi:170 [INFO] [multi] job_id 16896402 proc_lsst.multi INFO: [multi] job_id 16896402 2024-03-07 15:06:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 15:06:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896402 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896402 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16896402 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-07 15:06:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-07 15:06:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 15:06:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1232 try 0 launched on executor multi with executor id 1233 parsl.dataflow.dflow INFO: Standard output for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c1650ccb-71b3-43ce-a12c-e81da83fcbba_isrForCrosstalkSources_1125904_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c1650ccb-71b3-43ce-a12c-e81da83fcbba_isrForCrosstalkSources_1125904_6.stderr parsl.dataflow.dflow DEBUG: Task 1233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1233 has memoization hash 0bf4677b9d5c421161e95244644d187b parsl.dataflow.memoization INFO: Task 1233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ee3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1233 try 0 launched on executor multi with executor id 1234 parsl.dataflow.dflow INFO: Standard output for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/2f89dd3b-8364-484b-8fa7-1394f33447fa_isrForCrosstalkSources_1125893_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/2f89dd3b-8364-484b-8fa7-1394f33447fa_isrForCrosstalkSources_1125893_3.stderr parsl.dataflow.dflow DEBUG: Task 1234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1234 has memoization hash f8c2df9d31c7a409768fda143144f6e0 parsl.dataflow.memoization INFO: Task 1234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1234 try 0 launched on executor multi with executor id 1235 parsl.dataflow.dflow INFO: Standard output for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/4a7ec24d-1f1c-4db9-b2b2-5d77bb147c86_isrForCrosstalkSources_1125942_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/4a7ec24d-1f1c-4db9-b2b2-5d77bb147c86_isrForCrosstalkSources_1125942_27.stderr parsl.dataflow.dflow DEBUG: Task 1235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1235 has memoization hash c4c0c4efd77af3012ab604417129f247 parsl.dataflow.memoization INFO: Task 1235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1235 try 0 launched on executor multi with executor id 1236 parsl.dataflow.dflow INFO: Standard output for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/b592ffee-ef4f-499c-80f3-cb8e2c565d57_isrForCrosstalkSources_1125901_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/b592ffee-ef4f-499c-80f3-cb8e2c565d57_isrForCrosstalkSources_1125901_1.stderr parsl.dataflow.dflow DEBUG: Task 1236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1236 has memoization hash a4372176920d991de4ac5622c8e30c13 parsl.dataflow.memoization INFO: Task 1236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1236 try 0 launched on executor multi with executor id 1237 parsl.dataflow.dflow INFO: Standard output for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/ae0f9712-999c-477f-96a0-8933821d2e42_isrForCrosstalkSources_1125910_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/ae0f9712-999c-477f-96a0-8933821d2e42_isrForCrosstalkSources_1125910_34.stderr parsl.dataflow.dflow DEBUG: Task 1237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1237 has memoization hash 7877e8feb3d23be4041858962f856291 parsl.dataflow.memoization INFO: Task 1237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1237 try 0 launched on executor multi with executor id 1238 parsl.dataflow.dflow INFO: Standard output for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/a7ee5a15-3980-4212-aa16-49171e59f9ac_isrForCrosstalkSources_1125897_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/a7ee5a15-3980-4212-aa16-49171e59f9ac_isrForCrosstalkSources_1125897_4.stderr parsl.dataflow.dflow DEBUG: Task 1238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1238 has memoization hash 3f678a7e6bb0d71b575ba9fe898893a4 parsl.dataflow.memoization INFO: Task 1238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1238 try 0 launched on executor multi with executor id 1239 parsl.dataflow.dflow INFO: Standard output for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/6609779a-0470-4c02-baaf-b8d971f8aa3f_isrForCrosstalkSources_1125926_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/6609779a-0470-4c02-baaf-b8d971f8aa3f_isrForCrosstalkSources_1125926_7.stderr parsl.dataflow.dflow DEBUG: Task 1239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1239 has memoization hash fe59ee3b9804d121181656bb771ff00d parsl.dataflow.memoization INFO: Task 1239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1239 try 0 launched on executor multi with executor id 1240 parsl.dataflow.dflow INFO: Standard output for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/c97a2607-bb5a-435e-b07a-4a31a8e3af48_isrForCrosstalkSources_1125924_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/c97a2607-bb5a-435e-b07a-4a31a8e3af48_isrForCrosstalkSources_1125924_15.stderr parsl.dataflow.dflow DEBUG: Task 1240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1240 has memoization hash 1929c2297f068375d5cf7daf0b350965 parsl.dataflow.memoization INFO: Task 1240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1240 try 0 launched on executor multi with executor id 1241 parsl.dataflow.dflow INFO: Standard output for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8224e884-ce2a-4976-98c7-00fb4b3efe99_isrForCrosstalkSources_1125944_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8224e884-ce2a-4976-98c7-00fb4b3efe99_isrForCrosstalkSources_1125944_23.stderr parsl.dataflow.dflow DEBUG: Task 1241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1241 has memoization hash f54d06ecefb4e33f371fdfcf6a7f10ce parsl.dataflow.memoization INFO: Task 1241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1241 try 0 launched on executor multi with executor id 1242 parsl.dataflow.dflow INFO: Standard output for task 1241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/525d2620-eb2e-4a7b-b3e8-22abdfc95680_isrForCrosstalkSources_1125947_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/525d2620-eb2e-4a7b-b3e8-22abdfc95680_isrForCrosstalkSources_1125947_52.stderr parsl.dataflow.dflow DEBUG: Task 1242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1242 has memoization hash 4f37323c16b92ef65487cdc1e6c848bb parsl.dataflow.memoization INFO: Task 1242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1242 try 0 launched on executor multi with executor id 1243 parsl.dataflow.dflow INFO: Standard output for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/a11b7ffd-a296-454d-b9bf-610a311dbfde_isrForCrosstalkSources_1125908_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/a11b7ffd-a296-454d-b9bf-610a311dbfde_isrForCrosstalkSources_1125908_13.stderr parsl.dataflow.dflow DEBUG: Task 1243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1243 has memoization hash 124601488abe5ae34462fecb31be2914 parsl.dataflow.memoization INFO: Task 1243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1243 try 0 launched on executor multi with executor id 1244 parsl.dataflow.dflow INFO: Standard output for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/ab560e94-3483-48a7-bb2f-509fb421c510_isrForCrosstalkSources_1125907_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/ab560e94-3483-48a7-bb2f-509fb421c510_isrForCrosstalkSources_1125907_13.stderr parsl.dataflow.dflow DEBUG: Task 1244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1244 has memoization hash c13044e439f6ef99763bfed1a9aa9a46 parsl.dataflow.memoization INFO: Task 1244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1244 try 0 launched on executor multi with executor id 1245 parsl.dataflow.dflow INFO: Standard output for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/d7a2087e-fd24-4d30-85ba-87ac9053c0bf_isrForCrosstalkSources_1125903_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/d7a2087e-fd24-4d30-85ba-87ac9053c0bf_isrForCrosstalkSources_1125903_47.stderr parsl.dataflow.dflow DEBUG: Task 1245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1245 has memoization hash 8a7ac37cf69b0f6cb2f63b9b7a9c0f22 parsl.dataflow.memoization INFO: Task 1245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1245 try 0 launched on executor multi with executor id 1246 parsl.dataflow.dflow INFO: Standard output for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/b41d586a-32bc-405c-87e4-e6bafb5da769_isrForCrosstalkSources_1125939_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/b41d586a-32bc-405c-87e4-e6bafb5da769_isrForCrosstalkSources_1125939_52.stderr parsl.dataflow.dflow DEBUG: Task 1246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1246 has memoization hash 090b2af441a6d5ece9b9c3f1bb48bca9 parsl.dataflow.memoization INFO: Task 1246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1246 try 0 launched on executor multi with executor id 1247 parsl.dataflow.dflow INFO: Standard output for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/4e6bca42-9488-4281-a1d0-bc45b877565e_isrForCrosstalkSources_1125919_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/4e6bca42-9488-4281-a1d0-bc45b877565e_isrForCrosstalkSources_1125919_32.stderr parsl.dataflow.dflow DEBUG: Task 1247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1247 has memoization hash ee21e1586044923bafaecbbd5f9188c2 parsl.dataflow.memoization INFO: Task 1247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1247 try 0 launched on executor multi with executor id 1248 parsl.dataflow.dflow INFO: Standard output for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/007f31f0-0b9b-4862-8a7c-89389763a76a_isrForCrosstalkSources_1125908_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/007f31f0-0b9b-4862-8a7c-89389763a76a_isrForCrosstalkSources_1125908_35.stderr parsl.dataflow.dflow DEBUG: Task 1248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1248 has memoization hash 738212971817ccb048355c1454fe3a6c parsl.dataflow.memoization INFO: Task 1248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1248 try 0 launched on executor multi with executor id 1249 parsl.dataflow.dflow INFO: Standard output for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/cfd79448-bef9-4173-9526-180af367beb1_isrForCrosstalkSources_1125896_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/cfd79448-bef9-4173-9526-180af367beb1_isrForCrosstalkSources_1125896_22.stderr parsl.dataflow.dflow DEBUG: Task 1249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1249 has memoization hash 9225e0d82848bf232b5759547a9e72e0 parsl.dataflow.memoization INFO: Task 1249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1249 try 0 launched on executor multi with executor id 1250 parsl.dataflow.dflow INFO: Standard output for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/b3eb25b2-1393-4c06-a739-dbc47ccbca1f_isrForCrosstalkSources_1125912_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/b3eb25b2-1393-4c06-a739-dbc47ccbca1f_isrForCrosstalkSources_1125912_8.stderr parsl.dataflow.dflow DEBUG: Task 1250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1250 has memoization hash 9ea6cd59ecfef1c478f690eddcba616b parsl.dataflow.memoization INFO: Task 1250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1250 try 0 launched on executor multi with executor id 1251 parsl.dataflow.dflow INFO: Standard output for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/0dfc4cdf-eec4-4fa5-b2ab-e680dc638d56_isrForCrosstalkSources_1125923_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/0dfc4cdf-eec4-4fa5-b2ab-e680dc638d56_isrForCrosstalkSources_1125923_37.stderr parsl.dataflow.dflow DEBUG: Task 1251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1251 has memoization hash 8c29accc073e0875e4ebc06baa4613f0 parsl.dataflow.memoization INFO: Task 1251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1251 try 0 launched on executor multi with executor id 1252 parsl.dataflow.dflow INFO: Standard output for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/b3bc44fa-c280-4de7-9085-b2e38d575b1e_isrForCrosstalkSources_1125908_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/b3bc44fa-c280-4de7-9085-b2e38d575b1e_isrForCrosstalkSources_1125908_6.stderr parsl.dataflow.dflow DEBUG: Task 1252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1252 has memoization hash aea236503a45e2ace9c66d8245f8e0b6 parsl.dataflow.memoization INFO: Task 1252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:47 proc_lsst.multi:170 [INFO] [multi] job_id 16896403 proc_lsst.multi INFO: [multi] job_id 16896403 2024-03-07 15:06:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 15:06:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896403 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896403 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16896403 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-07 15:06:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-07 15:06:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 15:06:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1252 try 0 launched on executor multi with executor id 1253 parsl.dataflow.dflow INFO: Standard output for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/6b534939-c4c7-44ac-b61a-e990eb0a8508_isrForCrosstalkSources_1125913_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/6b534939-c4c7-44ac-b61a-e990eb0a8508_isrForCrosstalkSources_1125913_16.stderr parsl.dataflow.dflow DEBUG: Task 1253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1253 has memoization hash 4a28ab0429b0564a460c936264cc29c6 parsl.dataflow.memoization INFO: Task 1253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ed120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1253 try 0 launched on executor multi with executor id 1254 parsl.dataflow.dflow INFO: Standard output for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/67e1e236-da9a-4946-8c6b-a239bca9803c_isrForCrosstalkSources_1125888_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/67e1e236-da9a-4946-8c6b-a239bca9803c_isrForCrosstalkSources_1125888_61.stderr parsl.dataflow.dflow DEBUG: Task 1254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1254 has memoization hash ef3548ce94b9a3ecf2e2a23f0889e372 parsl.dataflow.memoization INFO: Task 1254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0edbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1254 try 0 launched on executor multi with executor id 1255 parsl.dataflow.dflow INFO: Standard output for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/4e996b9c-2c3d-442a-8cf6-4d2a57652923_isrForCrosstalkSources_1125936_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/4e996b9c-2c3d-442a-8cf6-4d2a57652923_isrForCrosstalkSources_1125936_27.stderr parsl.dataflow.dflow DEBUG: Task 1255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1255 has memoization hash dc77ba1fb8ea0923144d44477d53ddeb parsl.dataflow.memoization INFO: Task 1255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1255 try 0 launched on executor multi with executor id 1256 parsl.dataflow.dflow INFO: Standard output for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/59542692-4c93-4250-b2b2-9784f31257f2_isrForCrosstalkSources_1125929_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/59542692-4c93-4250-b2b2-9784f31257f2_isrForCrosstalkSources_1125929_31.stderr parsl.dataflow.dflow DEBUG: Task 1256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1256 has memoization hash 42d5089a9cbb54d4729471adf7e11f74 parsl.dataflow.memoization INFO: Task 1256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1256 try 0 launched on executor multi with executor id 1257 parsl.dataflow.dflow INFO: Standard output for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/45358667-cc37-486e-8948-8e8d593ac70a_isrForCrosstalkSources_1125936_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/45358667-cc37-486e-8948-8e8d593ac70a_isrForCrosstalkSources_1125936_55.stderr parsl.dataflow.dflow DEBUG: Task 1257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1257 has memoization hash 989c472cd53157e11f10c8722c383b57 parsl.dataflow.memoization INFO: Task 1257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1257 try 0 launched on executor multi with executor id 1258 parsl.dataflow.dflow INFO: Standard output for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/8a596052-9723-4e25-b219-5db29ff75940_isrForCrosstalkSources_1125915_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/8a596052-9723-4e25-b219-5db29ff75940_isrForCrosstalkSources_1125915_60.stderr parsl.dataflow.dflow DEBUG: Task 1258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1258 has memoization hash 9236b725ce2b94d44fece75c13e34957 parsl.dataflow.memoization INFO: Task 1258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1258 try 0 launched on executor multi with executor id 1259 parsl.dataflow.dflow INFO: Standard output for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/83d4c87e-3c5c-48ca-8fba-ff63cc17c41c_isrForCrosstalkSources_1125893_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/83d4c87e-3c5c-48ca-8fba-ff63cc17c41c_isrForCrosstalkSources_1125893_35.stderr parsl.dataflow.dflow DEBUG: Task 1259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1259 has memoization hash 9f0c107a197ab55466927d773ea234fa parsl.dataflow.memoization INFO: Task 1259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1259 try 0 launched on executor multi with executor id 1260 parsl.dataflow.dflow INFO: Standard output for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e169e421-eebf-4e29-ba26-76b5fc18b90c_isrForCrosstalkSources_1125942_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e169e421-eebf-4e29-ba26-76b5fc18b90c_isrForCrosstalkSources_1125942_57.stderr parsl.dataflow.dflow DEBUG: Task 1260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1260 has memoization hash d1e3a8dd8ce05c7f16e038df1c69a4d1 parsl.dataflow.memoization INFO: Task 1260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1260 try 0 launched on executor multi with executor id 1261 parsl.dataflow.dflow INFO: Standard output for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/da79051c-b6d7-4ccd-bfcf-0fcd8adba3ed_isrForCrosstalkSources_1125927_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/da79051c-b6d7-4ccd-bfcf-0fcd8adba3ed_isrForCrosstalkSources_1125927_55.stderr parsl.dataflow.dflow DEBUG: Task 1261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1261 has memoization hash 7fab7cfb5c223bbd4f0ef8f26f5cea26 parsl.dataflow.memoization INFO: Task 1261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1261 try 0 launched on executor multi with executor id 1262 parsl.dataflow.dflow INFO: Standard output for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/337d51a1-476b-4737-adfa-3bc77e9d30d0_isrForCrosstalkSources_1125924_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/337d51a1-476b-4737-adfa-3bc77e9d30d0_isrForCrosstalkSources_1125924_55.stderr parsl.dataflow.dflow DEBUG: Task 1262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1262 has memoization hash 816552478c4e8b56e568f5e4ae621f0f parsl.dataflow.memoization INFO: Task 1262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1262 try 0 launched on executor multi with executor id 1263 parsl.dataflow.dflow INFO: Standard output for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/feafcb43-9ba4-45ff-813f-ca3c20eafa61_isrForCrosstalkSources_1125892_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/feafcb43-9ba4-45ff-813f-ca3c20eafa61_isrForCrosstalkSources_1125892_21.stderr parsl.dataflow.dflow DEBUG: Task 1263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1263 has memoization hash ea219e6d57db26340d164ce5d9150d7a parsl.dataflow.memoization INFO: Task 1263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1263 try 0 launched on executor multi with executor id 1264 parsl.dataflow.dflow INFO: Standard output for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/95bf0aad-63a2-4998-93c2-7dde14b09db9_isrForCrosstalkSources_1125909_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/95bf0aad-63a2-4998-93c2-7dde14b09db9_isrForCrosstalkSources_1125909_25.stderr parsl.dataflow.dflow DEBUG: Task 1264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1264 has memoization hash 3da97d2029ff7af1026856d3514ef72a parsl.dataflow.memoization INFO: Task 1264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1264 try 0 launched on executor multi with executor id 1265 parsl.dataflow.dflow INFO: Standard output for task 1264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/c4861fa5-4cfd-4021-8162-f9123050e969_isrForCrosstalkSources_1125931_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/c4861fa5-4cfd-4021-8162-f9123050e969_isrForCrosstalkSources_1125931_7.stderr parsl.dataflow.dflow DEBUG: Task 1265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1265 has memoization hash f8aeb181b45e78bfc52d5d1bd9afca4f parsl.dataflow.memoization INFO: Task 1265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1265 try 0 launched on executor multi with executor id 1266 parsl.dataflow.dflow INFO: Standard output for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c1f2de04-24dc-4dff-8dad-42bfca12ba7c_isrForCrosstalkSources_1125904_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c1f2de04-24dc-4dff-8dad-42bfca12ba7c_isrForCrosstalkSources_1125904_5.stderr parsl.dataflow.dflow DEBUG: Task 1266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1266 has memoization hash 74395487302bba76866b55d4db30f589 parsl.dataflow.memoization INFO: Task 1266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1266 try 0 launched on executor multi with executor id 1267 parsl.dataflow.dflow INFO: Standard output for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/1a6aae29-e0ea-4255-83a4-00fd3942d7f6_isrForCrosstalkSources_1125903_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/1a6aae29-e0ea-4255-83a4-00fd3942d7f6_isrForCrosstalkSources_1125903_29.stderr parsl.dataflow.dflow DEBUG: Task 1267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1267 has memoization hash 358ec3cbf547d163e0b1f29e72c2d169 parsl.dataflow.memoization INFO: Task 1267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1267 try 0 launched on executor multi with executor id 1268 parsl.dataflow.dflow INFO: Standard output for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/03fb2389-9231-4e0d-88c8-4ffb8067da3f_isrForCrosstalkSources_1125938_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/03fb2389-9231-4e0d-88c8-4ffb8067da3f_isrForCrosstalkSources_1125938_50.stderr parsl.dataflow.dflow DEBUG: Task 1268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1268 has memoization hash be0caf07cae1c6061704855ffcb24826 parsl.dataflow.memoization INFO: Task 1268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1268 try 0 launched on executor multi with executor id 1269 parsl.dataflow.dflow INFO: Standard output for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/c31f6900-87dc-433b-8c93-8230a3fbb0aa_isrForCrosstalkSources_1125921_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/c31f6900-87dc-433b-8c93-8230a3fbb0aa_isrForCrosstalkSources_1125921_47.stderr parsl.dataflow.dflow DEBUG: Task 1269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1269 has memoization hash 25a77b6d8777ab2573b54325a013dce9 parsl.dataflow.memoization INFO: Task 1269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1269 try 0 launched on executor multi with executor id 1270 parsl.dataflow.dflow INFO: Standard output for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/21e9cf96-a605-4281-a843-c05f3f8d16a3_isrForCrosstalkSources_1125937_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/21e9cf96-a605-4281-a843-c05f3f8d16a3_isrForCrosstalkSources_1125937_12.stderr parsl.dataflow.dflow DEBUG: Task 1270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1270 has memoization hash 0034b796d8c67faa3c8837ea1589b8e1 parsl.dataflow.memoization INFO: Task 1270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1270 try 0 launched on executor multi with executor id 1271 parsl.dataflow.dflow INFO: Standard output for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/16c183e1-0a16-4065-9917-ee0ff7ab4789_isrForCrosstalkSources_1125899_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/16c183e1-0a16-4065-9917-ee0ff7ab4789_isrForCrosstalkSources_1125899_29.stderr parsl.dataflow.dflow DEBUG: Task 1271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1271 has memoization hash 8c0b1b63f94ab3f8fea17d79d5c87138 parsl.dataflow.memoization INFO: Task 1271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1271 try 0 launched on executor multi with executor id 1272 parsl.dataflow.dflow INFO: Standard output for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e6f2b599-4888-4623-bfc6-aa2fd27b9968_isrForCrosstalkSources_1125942_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e6f2b599-4888-4623-bfc6-aa2fd27b9968_isrForCrosstalkSources_1125942_16.stderr parsl.dataflow.dflow DEBUG: Task 1272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1272 has memoization hash 375ddd9f501217a4b1e2e49f3cd23f16 parsl.dataflow.memoization INFO: Task 1272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1272 try 0 launched on executor multi with executor id 1273 parsl.dataflow.dflow INFO: Standard output for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/4975aa27-b14e-491d-8aba-beba6d3481c5_isrForCrosstalkSources_1125944_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/4975aa27-b14e-491d-8aba-beba6d3481c5_isrForCrosstalkSources_1125944_11.stderr parsl.dataflow.dflow DEBUG: Task 1273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1273 has memoization hash a55c0b2f20b2565356b3f1f7d3f1a814 parsl.dataflow.memoization INFO: Task 1273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:47 proc_lsst.multi:170 [INFO] [multi] job_id 16896404 proc_lsst.multi INFO: [multi] job_id 16896404 2024-03-07 15:06:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 15:06:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896404 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896404 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16896404 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-07 15:06:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-07 15:06:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 15:06:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1273 try 0 launched on executor multi with executor id 1274 parsl.dataflow.dflow INFO: Standard output for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/409eb1b6-ddce-477a-9b51-f148330f4ede_isrForCrosstalkSources_1125942_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/409eb1b6-ddce-477a-9b51-f148330f4ede_isrForCrosstalkSources_1125942_28.stderr parsl.dataflow.dflow DEBUG: Task 1274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1274 has memoization hash 860c94584abb55e9493aadb19021ec66 parsl.dataflow.memoization INFO: Task 1274 had no result in cache parsl.providers.slurm.slurm DEBUG: moving files parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1274 try 0 launched on executor multi with executor id 1275 parsl.dataflow.dflow INFO: Standard output for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/413baf6e-cfe2-41af-a827-21f330507d39_isrForCrosstalkSources_1125943_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/413baf6e-cfe2-41af-a827-21f330507d39_isrForCrosstalkSources_1125943_58.stderr parsl.dataflow.dflow DEBUG: Task 1275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1275 has memoization hash 0c45d0220f690082b8e980de44380235 parsl.dataflow.memoization INFO: Task 1275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1275 try 0 launched on executor multi with executor id 1276 parsl.dataflow.dflow INFO: Standard output for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/9056c9d5-b812-4241-ba39-3d9ada7d7895_isrForCrosstalkSources_1125917_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/9056c9d5-b812-4241-ba39-3d9ada7d7895_isrForCrosstalkSources_1125917_22.stderr parsl.dataflow.dflow DEBUG: Task 1276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1276 has memoization hash bc29648440cb321ff3ae9ec98cd81c51 parsl.dataflow.memoization INFO: Task 1276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1276 try 0 launched on executor multi with executor id 1277 parsl.dataflow.dflow INFO: Standard output for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/ff008800-7982-4ad8-8956-59c4aeead3c5_isrForCrosstalkSources_1125934_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/ff008800-7982-4ad8-8956-59c4aeead3c5_isrForCrosstalkSources_1125934_44.stderr parsl.dataflow.dflow DEBUG: Task 1277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1277 has memoization hash 921d092812d6ecc07633fba9f7c8cd4f parsl.dataflow.memoization INFO: Task 1277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ed300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1277 try 0 launched on executor multi with executor id 1278 parsl.dataflow.dflow INFO: Standard output for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/ded33ae2-c238-409b-8189-98fa213185b7_isrForCrosstalkSources_1125908_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/ded33ae2-c238-409b-8189-98fa213185b7_isrForCrosstalkSources_1125908_31.stderr parsl.dataflow.dflow DEBUG: Task 1278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1278 has memoization hash cb0d3112de25f4ac2b0436a9543be212 parsl.dataflow.memoization INFO: Task 1278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1278 try 0 launched on executor multi with executor id 1279 parsl.dataflow.dflow INFO: Standard output for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/4587f0a3-6d09-4258-a6ae-431ecd7ff800_isrForCrosstalkSources_1125906_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/4587f0a3-6d09-4258-a6ae-431ecd7ff800_isrForCrosstalkSources_1125906_14.stderr parsl.dataflow.dflow DEBUG: Task 1279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1279 has memoization hash f2fcd971ec207385973e3c285321e761 parsl.dataflow.memoization INFO: Task 1279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1279 try 0 launched on executor multi with executor id 1280 parsl.dataflow.dflow INFO: Standard output for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/2c16d3a2-4fbf-43fd-b024-3be17a4f0adf_isrForCrosstalkSources_1125889_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/2c16d3a2-4fbf-43fd-b024-3be17a4f0adf_isrForCrosstalkSources_1125889_34.stderr parsl.dataflow.dflow DEBUG: Task 1280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1280 has memoization hash 0e2620209a12d96e6eab57d5e9c69e85 parsl.dataflow.memoization INFO: Task 1280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1280 try 0 launched on executor multi with executor id 1281 parsl.dataflow.dflow INFO: Standard output for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/c9c58b0f-8bfc-4a00-824d-027fa62a3aa5_isrForCrosstalkSources_1125939_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/c9c58b0f-8bfc-4a00-824d-027fa62a3aa5_isrForCrosstalkSources_1125939_61.stderr parsl.dataflow.dflow DEBUG: Task 1281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1281 has memoization hash ae24937b6fc30e2183622bb3328d313a parsl.dataflow.memoization INFO: Task 1281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1281 try 0 launched on executor multi with executor id 1282 parsl.dataflow.dflow INFO: Standard output for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/96a5d255-bb66-46be-b6c6-5d5aa295d4c4_isrForCrosstalkSources_1125900_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/96a5d255-bb66-46be-b6c6-5d5aa295d4c4_isrForCrosstalkSources_1125900_30.stderr parsl.dataflow.dflow DEBUG: Task 1282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1282 has memoization hash 094c79543268fb9815de9923890f6f90 parsl.dataflow.memoization INFO: Task 1282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1282 try 0 launched on executor multi with executor id 1283 parsl.dataflow.dflow INFO: Standard output for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/060946aa-457a-4f50-baa3-00f0469ff05b_isrForCrosstalkSources_1125901_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/060946aa-457a-4f50-baa3-00f0469ff05b_isrForCrosstalkSources_1125901_49.stderr parsl.dataflow.dflow DEBUG: Task 1283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1283 has memoization hash 31abb85b85c9de620729ff76c0c425fb parsl.dataflow.memoization INFO: Task 1283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1283 try 0 launched on executor multi with executor id 1284 parsl.dataflow.dflow INFO: Standard output for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/47d78eaf-9621-437e-bcc4-a8bb97478edd_isrForCrosstalkSources_1125915_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/47d78eaf-9621-437e-bcc4-a8bb97478edd_isrForCrosstalkSources_1125915_36.stderr parsl.dataflow.dflow DEBUG: Task 1284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1284 has memoization hash c64b720ab7cb502cde74979c35a9feb0 parsl.dataflow.memoization INFO: Task 1284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1284 try 0 launched on executor multi with executor id 1285 parsl.dataflow.dflow INFO: Standard output for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/efd5e822-54f5-427d-904b-929451dd16bc_isrForCrosstalkSources_1125943_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/efd5e822-54f5-427d-904b-929451dd16bc_isrForCrosstalkSources_1125943_28.stderr parsl.dataflow.dflow DEBUG: Task 1285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1285 has memoization hash 0dd5f2863b2c99a9579e25645a676f12 parsl.dataflow.memoization INFO: Task 1285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1285 try 0 launched on executor multi with executor id 1286 parsl.dataflow.dflow INFO: Standard output for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/2a3360ca-bf6a-4902-aaa9-1c17177ca2c7_isrForCrosstalkSources_1125941_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/2a3360ca-bf6a-4902-aaa9-1c17177ca2c7_isrForCrosstalkSources_1125941_62.stderr parsl.dataflow.dflow DEBUG: Task 1286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1286 has memoization hash 7702d4e50b3856642dc483e0fcd38b83 parsl.dataflow.memoization INFO: Task 1286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1286 try 0 launched on executor multi with executor id 1287 parsl.dataflow.dflow INFO: Standard output for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/d06956f4-38ee-4b18-9701-607a187bf1f0_isrForCrosstalkSources_1125894_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/d06956f4-38ee-4b18-9701-607a187bf1f0_isrForCrosstalkSources_1125894_17.stderr parsl.dataflow.dflow DEBUG: Task 1287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1287 has memoization hash 68e7f3a5ae94b6588a8c49747969f17d parsl.dataflow.memoization INFO: Task 1287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1287 try 0 launched on executor multi with executor id 1288 parsl.dataflow.dflow INFO: Standard output for task 1287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/5f519bc6-1370-40ed-bac9-3728eff84364_isrForCrosstalkSources_1125917_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/5f519bc6-1370-40ed-bac9-3728eff84364_isrForCrosstalkSources_1125917_2.stderr parsl.dataflow.dflow DEBUG: Task 1288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1288 has memoization hash 3d8449503ff5318c18bae622daa72540 parsl.dataflow.memoization INFO: Task 1288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1288 try 0 launched on executor multi with executor id 1289 parsl.dataflow.dflow INFO: Standard output for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/e7815104-22d4-41f0-9ae6-64900868bb2d_isrForCrosstalkSources_1125906_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/e7815104-22d4-41f0-9ae6-64900868bb2d_isrForCrosstalkSources_1125906_62.stderr parsl.dataflow.dflow DEBUG: Task 1289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1289 has memoization hash 16d11c760feb8b1c65ff0641e4d5a73c parsl.dataflow.memoization INFO: Task 1289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1289 try 0 launched on executor multi with executor id 1290 parsl.dataflow.dflow INFO: Standard output for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/739fcc1e-a0ba-440c-803f-71926b290a6b_isrForCrosstalkSources_1125932_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/739fcc1e-a0ba-440c-803f-71926b290a6b_isrForCrosstalkSources_1125932_35.stderr parsl.dataflow.dflow DEBUG: Task 1290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1290 has memoization hash c702599e6777d96446dbfe258a0a289a parsl.dataflow.memoization INFO: Task 1290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1290 try 0 launched on executor multi with executor id 1291 parsl.dataflow.dflow INFO: Standard output for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/7fff316d-d714-4de8-82bf-a738c7cadb87_isrForCrosstalkSources_1125944_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/7fff316d-d714-4de8-82bf-a738c7cadb87_isrForCrosstalkSources_1125944_56.stderr parsl.dataflow.dflow DEBUG: Task 1291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1291 has memoization hash 775f9e41f55977327eb45a6cf1b130a5 parsl.dataflow.memoization INFO: Task 1291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1291 try 0 launched on executor multi with executor id 1292 parsl.dataflow.dflow INFO: Standard output for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a03a9713-7b3c-4d97-b32d-e45e82c49e10_isrForCrosstalkSources_1125937_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a03a9713-7b3c-4d97-b32d-e45e82c49e10_isrForCrosstalkSources_1125937_35.stderr parsl.dataflow.dflow DEBUG: Task 1292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1292 has memoization hash e3108cc951734a484dacb0c45f1613db parsl.dataflow.memoization INFO: Task 1292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1292 try 0 launched on executor multi with executor id 1293 parsl.dataflow.dflow INFO: Standard output for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/50bc19b7-ad86-47a8-a0fb-6f3ff2038d9c_isrForCrosstalkSources_1125919_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/50bc19b7-ad86-47a8-a0fb-6f3ff2038d9c_isrForCrosstalkSources_1125919_17.stderr parsl.dataflow.dflow DEBUG: Task 1293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1293 has memoization hash f5b14a3b326ead06ef2a97f52d3b2bfc parsl.dataflow.memoization INFO: Task 1293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1293 try 0 launched on executor multi with executor id 1294 parsl.dataflow.dflow INFO: Standard output for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6187af46-11da-41cc-b0dc-77158b1b3c79_isrForCrosstalkSources_1125927_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6187af46-11da-41cc-b0dc-77158b1b3c79_isrForCrosstalkSources_1125927_44.stderr parsl.dataflow.dflow DEBUG: Task 1294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1294 has memoization hash 44f31f15c1b5330ce0482dfaa68e4dcc parsl.dataflow.memoization INFO: Task 1294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1294 try 0 launched on executor multi with executor id 1295 parsl.dataflow.dflow INFO: Standard output for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/87ecce0c-27f2-408c-bcb6-f16042da2671_isrForCrosstalkSources_1125903_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/87ecce0c-27f2-408c-bcb6-f16042da2671_isrForCrosstalkSources_1125903_33.stderr parsl.dataflow.dflow DEBUG: Task 1295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1295 has memoization hash f40fbaf61feeb9daaf67d2c95475adaa parsl.dataflow.memoization INFO: Task 1295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1295 try 0 launched on executor multi with executor id 1296 parsl.dataflow.dflow INFO: Standard output for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7bfcbab8-e1d0-4d30-87a8-c10ee8644157_isrForCrosstalkSources_1125929_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7bfcbab8-e1d0-4d30-87a8-c10ee8644157_isrForCrosstalkSources_1125929_1.stderr parsl.dataflow.dflow DEBUG: Task 1296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1296 has memoization hash 018bb832b7e1fd21ccb606d1a4849dad parsl.dataflow.memoization INFO: Task 1296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1296 try 0 launched on executor multi with executor id 1297 parsl.dataflow.dflow INFO: Standard output for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/b4bab9df-e167-4cc7-86ed-e5804a29e492_isrForCrosstalkSources_1125916_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/b4bab9df-e167-4cc7-86ed-e5804a29e492_isrForCrosstalkSources_1125916_61.stderr parsl.dataflow.dflow DEBUG: Task 1297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1297 has memoization hash e619641032ee7f2892f03941e8b68d92 parsl.dataflow.memoization INFO: Task 1297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1297 try 0 launched on executor multi with executor id 1298 parsl.dataflow.dflow INFO: Standard output for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/23ed7ffb-ff1d-4ef2-9b27-14f977175360_isrForCrosstalkSources_1125900_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/23ed7ffb-ff1d-4ef2-9b27-14f977175360_isrForCrosstalkSources_1125900_32.stderr parsl.dataflow.dflow DEBUG: Task 1298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1298 has memoization hash bc304456c8bf05bf225f461d6263c39b parsl.dataflow.memoization INFO: Task 1298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1298 try 0 launched on executor multi with executor id 1299 parsl.dataflow.dflow INFO: Standard output for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/34e9d438-ab11-4dbb-92e7-316d09967d17_isrForCrosstalkSources_1125929_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/34e9d438-ab11-4dbb-92e7-316d09967d17_isrForCrosstalkSources_1125929_14.stderr parsl.dataflow.dflow DEBUG: Task 1299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1299 has memoization hash 91f9fe6f542953bc5bcbeea08340b78b parsl.dataflow.memoization INFO: Task 1299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1299 try 0 launched on executor multi with executor id 1300 parsl.dataflow.dflow INFO: Standard output for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/bacf9355-ac97-4d51-a2d6-de9ce4188045_isrForCrosstalkSources_1125943_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/bacf9355-ac97-4d51-a2d6-de9ce4188045_isrForCrosstalkSources_1125943_48.stderr parsl.dataflow.dflow DEBUG: Task 1300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1300 has memoization hash d30a18c82621759dad74533e7d7f7ad6 parsl.dataflow.memoization INFO: Task 1300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:48 proc_lsst.multi:170 [INFO] [multi] job_id 16896405 proc_lsst.multi INFO: [multi] job_id 16896405 2024-03-07 15:06:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 15:06:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896405 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896405 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16896405 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-07 15:06:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-07 15:06:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 15:06:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1300 try 0 launched on executor multi with executor id 1301 parsl.dataflow.dflow INFO: Standard output for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/8b2fc2ff-f68e-454d-83fc-24ef32de8074_isrForCrosstalkSources_1125914_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/8b2fc2ff-f68e-454d-83fc-24ef32de8074_isrForCrosstalkSources_1125914_15.stderr parsl.dataflow.dflow DEBUG: Task 1301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1301 has memoization hash 408d9ea08a8629f1b986d1182a6fd5d0 parsl.dataflow.memoization INFO: Task 1301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1301 try 0 launched on executor multi with executor id 1302 parsl.dataflow.dflow INFO: Standard output for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/a9534ca5-3b26-49d1-9372-6203a7b53c4a_isrForCrosstalkSources_1125929_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/a9534ca5-3b26-49d1-9372-6203a7b53c4a_isrForCrosstalkSources_1125929_22.stderr parsl.dataflow.dflow DEBUG: Task 1302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1302 has memoization hash 5097e00d7f3babd77a7f456759bd7070 parsl.dataflow.memoization INFO: Task 1302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1302 try 0 launched on executor multi with executor id 1303 parsl.dataflow.dflow INFO: Standard output for task 1302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/aa2855bc-764d-4bbe-a701-9ca87ead0ce5_isrForCrosstalkSources_1125946_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/aa2855bc-764d-4bbe-a701-9ca87ead0ce5_isrForCrosstalkSources_1125946_38.stderr parsl.dataflow.dflow DEBUG: Task 1303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1303 has memoization hash fda636459e37f616ca2b0218cd368aba parsl.dataflow.memoization INFO: Task 1303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1303 try 0 launched on executor multi with executor id 1304 parsl.dataflow.dflow INFO: Standard output for task 1303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/b572b8c6-58a5-496c-a52e-8d75acad1f72_isrForCrosstalkSources_1125940_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/b572b8c6-58a5-496c-a52e-8d75acad1f72_isrForCrosstalkSources_1125940_46.stderr parsl.dataflow.dflow DEBUG: Task 1304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1304 has memoization hash 8084b0e579730acc7c6c029e010a9414 parsl.dataflow.memoization INFO: Task 1304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1304 try 0 launched on executor multi with executor id 1305 parsl.dataflow.dflow INFO: Standard output for task 1304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/245a3ebc-1198-4ef0-a76a-9a7032ff337a_isrForCrosstalkSources_1125892_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/245a3ebc-1198-4ef0-a76a-9a7032ff337a_isrForCrosstalkSources_1125892_30.stderr parsl.dataflow.dflow DEBUG: Task 1305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1305 has memoization hash 61d8604e2d8af102d66b777687b14468 parsl.dataflow.memoization INFO: Task 1305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1305 try 0 launched on executor multi with executor id 1306 parsl.dataflow.dflow INFO: Standard output for task 1305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/2c813077-3455-488a-882d-efec149be6ad_isrForCrosstalkSources_1125921_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/2c813077-3455-488a-882d-efec149be6ad_isrForCrosstalkSources_1125921_41.stderr parsl.dataflow.dflow DEBUG: Task 1306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1306 has memoization hash 7f61ee0249e376edd8bd47cc9047b9a4 parsl.dataflow.memoization INFO: Task 1306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1306 try 0 launched on executor multi with executor id 1307 parsl.dataflow.dflow INFO: Standard output for task 1306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d179dadc-4384-4141-9c1c-353ee47ca0cb_isrForCrosstalkSources_1125911_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d179dadc-4384-4141-9c1c-353ee47ca0cb_isrForCrosstalkSources_1125911_15.stderr parsl.dataflow.dflow DEBUG: Task 1307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1307 has memoization hash d15a957ab0b2c047169bfad63a01c836 parsl.dataflow.memoization INFO: Task 1307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1307 try 0 launched on executor multi with executor id 1308 parsl.dataflow.dflow INFO: Standard output for task 1307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/cf47ce0b-fb02-4d39-987d-00d043798115_isrForCrosstalkSources_1125893_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/cf47ce0b-fb02-4d39-987d-00d043798115_isrForCrosstalkSources_1125893_28.stderr parsl.dataflow.dflow DEBUG: Task 1308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1308 has memoization hash 7a1d9337e320ce983baf1c4f1b3c803f parsl.dataflow.memoization INFO: Task 1308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1308 try 0 launched on executor multi with executor id 1309 parsl.dataflow.dflow INFO: Standard output for task 1308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/46800cc2-7aca-46bb-9703-c23cb06a5527_isrForCrosstalkSources_1125940_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/46800cc2-7aca-46bb-9703-c23cb06a5527_isrForCrosstalkSources_1125940_32.stderr parsl.dataflow.dflow DEBUG: Task 1309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1309 has memoization hash 2ccc9044f38d91899e3dd9059c7cbd88 parsl.dataflow.memoization INFO: Task 1309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1309 try 0 launched on executor multi with executor id 1310 parsl.dataflow.dflow INFO: Standard output for task 1309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/2f282985-f0d4-4287-9409-e6f99e1516ce_isrForCrosstalkSources_1125910_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/2f282985-f0d4-4287-9409-e6f99e1516ce_isrForCrosstalkSources_1125910_17.stderr parsl.dataflow.dflow DEBUG: Task 1310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1310 has memoization hash 86de9b88595b9c421231bb9d5fa8d0ba parsl.dataflow.memoization INFO: Task 1310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1310 try 0 launched on executor multi with executor id 1311 parsl.dataflow.dflow INFO: Standard output for task 1310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/17e2a5f8-5ecc-42d3-9735-84fa78973479_isrForCrosstalkSources_1125935_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/17e2a5f8-5ecc-42d3-9735-84fa78973479_isrForCrosstalkSources_1125935_2.stderr parsl.dataflow.dflow DEBUG: Task 1311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1311 has memoization hash cfad54a8a2d4ed1efd15072801b93c08 parsl.dataflow.memoization INFO: Task 1311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1311 try 0 launched on executor multi with executor id 1312 parsl.dataflow.dflow INFO: Standard output for task 1311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/ac729294-fa15-4b4b-888b-ae1b08487095_isrForCrosstalkSources_1125931_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/ac729294-fa15-4b4b-888b-ae1b08487095_isrForCrosstalkSources_1125931_4.stderr parsl.dataflow.dflow DEBUG: Task 1312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1312 has memoization hash dfe6346851c433ffe9e1609d87ba6206 parsl.dataflow.memoization INFO: Task 1312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1312 try 0 launched on executor multi with executor id 1313 parsl.dataflow.dflow INFO: Standard output for task 1312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/540ed04b-2dbf-437c-b9d4-c10aefc1af24_isrForCrosstalkSources_1125891_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/540ed04b-2dbf-437c-b9d4-c10aefc1af24_isrForCrosstalkSources_1125891_50.stderr parsl.dataflow.dflow DEBUG: Task 1313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1313 has memoization hash 1284a566ae8855c340e64b9cf67d49e9 parsl.dataflow.memoization INFO: Task 1313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1313 try 0 launched on executor multi with executor id 1314 parsl.dataflow.dflow INFO: Standard output for task 1313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/195deb70-0a45-4fcf-b84d-befc397b8c9d_isrForCrosstalkSources_1125947_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/195deb70-0a45-4fcf-b84d-befc397b8c9d_isrForCrosstalkSources_1125947_4.stderr parsl.dataflow.dflow DEBUG: Task 1314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1314 has memoization hash b43e15bbf5cc38af810da15932b9db17 parsl.dataflow.memoization INFO: Task 1314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1314 try 0 launched on executor multi with executor id 1315 parsl.dataflow.dflow INFO: Standard output for task 1314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/ff21f25c-c5e5-404b-a7f6-163018eb4886_isrForCrosstalkSources_1125921_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/ff21f25c-c5e5-404b-a7f6-163018eb4886_isrForCrosstalkSources_1125921_43.stderr parsl.dataflow.dflow DEBUG: Task 1315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1315 has memoization hash 0864b12201911a30f7bd16062e409b9e parsl.dataflow.memoization INFO: Task 1315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1315 try 0 launched on executor multi with executor id 1316 parsl.dataflow.dflow INFO: Standard output for task 1315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/bcf44c23-a51e-46cf-a447-7d2339dbf47c_isrForCrosstalkSources_1125942_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/bcf44c23-a51e-46cf-a447-7d2339dbf47c_isrForCrosstalkSources_1125942_5.stderr parsl.dataflow.dflow DEBUG: Task 1316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1316 has memoization hash 706d508c2a678b44f36a70329ddf561a parsl.dataflow.memoization INFO: Task 1316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1316 try 0 launched on executor multi with executor id 1317 parsl.dataflow.dflow INFO: Standard output for task 1316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/896c9626-b8b0-4d62-9dc9-a4c291d29ff3_isrForCrosstalkSources_1125917_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/896c9626-b8b0-4d62-9dc9-a4c291d29ff3_isrForCrosstalkSources_1125917_26.stderr parsl.dataflow.dflow DEBUG: Task 1317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1317 has memoization hash 4461a05d45950af14d7b23521f126979 parsl.dataflow.memoization INFO: Task 1317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1317 try 0 launched on executor multi with executor id 1318 parsl.dataflow.dflow INFO: Standard output for task 1317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/b899de79-06c0-45b0-bd3d-64dcd8223d7b_isrForCrosstalkSources_1125931_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/b899de79-06c0-45b0-bd3d-64dcd8223d7b_isrForCrosstalkSources_1125931_30.stderr parsl.dataflow.dflow DEBUG: Task 1318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1318 has memoization hash 8b2c3f8f218a3bf7758fc026b69451be parsl.dataflow.memoization INFO: Task 1318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1318 try 0 launched on executor multi with executor id 1319 parsl.dataflow.dflow INFO: Standard output for task 1318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/781b9ab1-79d7-47fb-9ee8-13d50d71ab39_isrForCrosstalkSources_1125945_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/781b9ab1-79d7-47fb-9ee8-13d50d71ab39_isrForCrosstalkSources_1125945_44.stderr parsl.dataflow.dflow DEBUG: Task 1319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1319 has memoization hash 10cf0f52ce77d79b20cc7fa4d5badd88 parsl.dataflow.memoization INFO: Task 1319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1319 try 0 launched on executor multi with executor id 1320 parsl.dataflow.dflow INFO: Standard output for task 1319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/6c029321-1cd4-4506-9663-0c0e6ea002eb_isrForCrosstalkSources_1125897_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/6c029321-1cd4-4506-9663-0c0e6ea002eb_isrForCrosstalkSources_1125897_59.stderr parsl.dataflow.dflow DEBUG: Task 1320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1320 has memoization hash de8a33b7248292dc44a1ade19d8f78f8 parsl.dataflow.memoization INFO: Task 1320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1320 try 0 launched on executor multi with executor id 1321 parsl.dataflow.dflow INFO: Standard output for task 1320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/8bd6d832-b78a-44ca-bcd8-45dfc71684c3_isrForCrosstalkSources_1125924_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/8bd6d832-b78a-44ca-bcd8-45dfc71684c3_isrForCrosstalkSources_1125924_50.stderr parsl.dataflow.dflow DEBUG: Task 1321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1321 has memoization hash 75188a0bffb87aeb4e0cf5faaaea39eb parsl.dataflow.memoization INFO: Task 1321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1321 try 0 launched on executor multi with executor id 1322 parsl.dataflow.dflow INFO: Standard output for task 1321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/b695cf2e-0add-4d04-bcbd-f932af6ae2e9_isrForCrosstalkSources_1125940_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/b695cf2e-0add-4d04-bcbd-f932af6ae2e9_isrForCrosstalkSources_1125940_12.stderr parsl.dataflow.dflow DEBUG: Task 1322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1322 has memoization hash f22caec0266f82c6fa7c88c21a857dcb parsl.dataflow.memoization INFO: Task 1322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1322 try 0 launched on executor multi with executor id 1323 parsl.dataflow.dflow INFO: Standard output for task 1322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/6eba5f12-1272-48d2-9096-3cc79e2ca48e_isrForCrosstalkSources_1125914_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/6eba5f12-1272-48d2-9096-3cc79e2ca48e_isrForCrosstalkSources_1125914_61.stderr parsl.dataflow.dflow DEBUG: Task 1323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1323 has memoization hash 5ff2c4a752e7ff4feffa26ebbe4d65da parsl.dataflow.memoization INFO: Task 1323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1323 try 0 launched on executor multi with executor id 1324 parsl.dataflow.dflow INFO: Standard output for task 1323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/8fa1c53c-0b47-48db-960d-1d860b276444_isrForCrosstalkSources_1125892_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/8fa1c53c-0b47-48db-960d-1d860b276444_isrForCrosstalkSources_1125892_17.stderr parsl.dataflow.dflow DEBUG: Task 1324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1324 has memoization hash 813f5ad59f98232990991002f7baf46a parsl.dataflow.memoization INFO: Task 1324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1324 try 0 launched on executor multi with executor id 1325 parsl.dataflow.dflow INFO: Standard output for task 1324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/88e35c44-4a4c-420d-b5d1-7f5082c75aba_isrForCrosstalkSources_1125903_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/88e35c44-4a4c-420d-b5d1-7f5082c75aba_isrForCrosstalkSources_1125903_9.stderr parsl.dataflow.dflow DEBUG: Task 1325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1325 has memoization hash c99b4922ad1aba323b95eb68240aa016 parsl.dataflow.memoization INFO: Task 1325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1325 try 0 launched on executor multi with executor id 1326 parsl.dataflow.dflow INFO: Standard output for task 1325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/2c5e06b6-7dc2-43a9-9afc-92810d476df3_isrForCrosstalkSources_1125889_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/2c5e06b6-7dc2-43a9-9afc-92810d476df3_isrForCrosstalkSources_1125889_40.stderr parsl.dataflow.dflow DEBUG: Task 1326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1326 has memoization hash f95bbf69a566054e2ed68703d3248c47 parsl.dataflow.memoization INFO: Task 1326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1326 try 0 launched on executor multi with executor id 1327 parsl.dataflow.dflow INFO: Standard output for task 1326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/427128e4-9b58-42ed-9599-2372ec1253cc_isrForCrosstalkSources_1125926_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/427128e4-9b58-42ed-9599-2372ec1253cc_isrForCrosstalkSources_1125926_19.stderr parsl.dataflow.dflow DEBUG: Task 1327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1327 has memoization hash f38691e6994c3b0c453cce6d8f8c6209 parsl.dataflow.memoization INFO: Task 1327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1327 try 0 launched on executor multi with executor id 1328 parsl.dataflow.dflow INFO: Standard output for task 1327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/d9b7f00f-ae5b-437a-bdd3-ec25a187ef3e_isrForCrosstalkSources_1125927_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/d9b7f00f-ae5b-437a-bdd3-ec25a187ef3e_isrForCrosstalkSources_1125927_59.stderr parsl.dataflow.dflow DEBUG: Task 1328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1328 has memoization hash 114cfb5c255af6d0a244bd7ab6cfcb8b parsl.dataflow.memoization INFO: Task 1328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1328 try 0 launched on executor multi with executor id 1329 parsl.dataflow.dflow INFO: Standard output for task 1328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/90a874a1-761d-453a-a161-4ea6d3970154_isrForCrosstalkSources_1125927_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/90a874a1-761d-453a-a161-4ea6d3970154_isrForCrosstalkSources_1125927_52.stderr parsl.dataflow.dflow DEBUG: Task 1329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1329 has memoization hash 368585fd6209b7eeb3cf680ddda4e4eb parsl.dataflow.memoization INFO: Task 1329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1329 try 0 launched on executor multi with executor id 1330 parsl.dataflow.dflow INFO: Standard output for task 1329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/1635a302-c05d-42d3-a86d-e163fd9434ae_isrForCrosstalkSources_1125935_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/1635a302-c05d-42d3-a86d-e163fd9434ae_isrForCrosstalkSources_1125935_9.stderr parsl.dataflow.dflow DEBUG: Task 1330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1330 has memoization hash 17ebffce429f1466609ed45c334ed72b parsl.dataflow.memoization INFO: Task 1330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1330 try 0 launched on executor multi with executor id 1331 parsl.dataflow.dflow INFO: Standard output for task 1330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/86d33459-69a8-4a57-9703-fda100fb9873_isrForCrosstalkSources_1125895_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/86d33459-69a8-4a57-9703-fda100fb9873_isrForCrosstalkSources_1125895_31.stderr parsl.dataflow.dflow DEBUG: Task 1331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1331 has memoization hash 1cfc8f296a76e26db5bee6208f8d08ee parsl.dataflow.memoization INFO: Task 1331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1331 try 0 launched on executor multi with executor id 1332 parsl.dataflow.dflow INFO: Standard output for task 1331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/28780cc2-6441-4e49-9530-dc725bb4d8fc_isrForCrosstalkSources_1125905_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/28780cc2-6441-4e49-9530-dc725bb4d8fc_isrForCrosstalkSources_1125905_19.stderr parsl.dataflow.dflow DEBUG: Task 1332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1332 has memoization hash 5a05f087ae8775b91811f84b06f8846f parsl.dataflow.memoization INFO: Task 1332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0ee7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:48 proc_lsst.multi:170 [INFO] [multi] job_id 16896408 proc_lsst.multi INFO: [multi] job_id 16896408 2024-03-07 15:06:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:06:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896408 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896408 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16896408 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-07 15:06:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-07 15:06:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:06:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1332 try 0 launched on executor multi with executor id 1333 parsl.dataflow.dflow INFO: Standard output for task 1332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/43e2d641-673e-476c-8e65-2261a70e6fde_isrForCrosstalkSources_1125924_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/43e2d641-673e-476c-8e65-2261a70e6fde_isrForCrosstalkSources_1125924_41.stderr parsl.dataflow.dflow DEBUG: Task 1333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1333 has memoization hash 6468a64254306ad0bf279cb8210e9f9e parsl.dataflow.memoization INFO: Task 1333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1333 try 0 launched on executor multi with executor id 1334 parsl.dataflow.dflow INFO: Standard output for task 1333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/0fcf56d1-1d1b-48fb-8a5f-2a91f80a0374_isrForCrosstalkSources_1125938_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/0fcf56d1-1d1b-48fb-8a5f-2a91f80a0374_isrForCrosstalkSources_1125938_31.stderr parsl.dataflow.dflow DEBUG: Task 1334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1334 has memoization hash b605145d5a88c1a772d34e357fcb52b6 parsl.dataflow.memoization INFO: Task 1334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1334 try 0 launched on executor multi with executor id 1335 parsl.dataflow.dflow INFO: Standard output for task 1334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/802b1dee-c621-46b8-9d44-732d2259b570_isrForCrosstalkSources_1125904_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/802b1dee-c621-46b8-9d44-732d2259b570_isrForCrosstalkSources_1125904_29.stderr parsl.dataflow.dflow DEBUG: Task 1335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1335 has memoization hash 9e9216fad4d4d47f51f01e80fe4e8365 parsl.dataflow.memoization INFO: Task 1335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1335 try 0 launched on executor multi with executor id 1336 parsl.dataflow.dflow INFO: Standard output for task 1335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/161413ea-00fc-4b81-867e-65ab0c228f47_isrForCrosstalkSources_1125925_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/161413ea-00fc-4b81-867e-65ab0c228f47_isrForCrosstalkSources_1125925_15.stderr parsl.dataflow.dflow DEBUG: Task 1336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1336 has memoization hash 86464ea74723313ad20a1bad9c22c411 parsl.dataflow.memoization INFO: Task 1336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1336 try 0 launched on executor multi with executor id 1337 parsl.dataflow.dflow INFO: Standard output for task 1336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/00d29b24-2f62-4806-b55f-ee7c086e6b25_isrForCrosstalkSources_1125927_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/00d29b24-2f62-4806-b55f-ee7c086e6b25_isrForCrosstalkSources_1125927_28.stderr parsl.dataflow.dflow DEBUG: Task 1337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1337 has memoization hash a63f060a90dbba8ec003a4cab08cb2f7 parsl.dataflow.memoization INFO: Task 1337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1337 try 0 launched on executor multi with executor id 1338 parsl.dataflow.dflow INFO: Standard output for task 1337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/b2e050fb-d997-4511-8650-e26f58d5c2a3_isrForCrosstalkSources_1125906_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/b2e050fb-d997-4511-8650-e26f58d5c2a3_isrForCrosstalkSources_1125906_38.stderr parsl.dataflow.dflow DEBUG: Task 1338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1338 has memoization hash b2fad23a5289e8b85263b0b5e0f6ffc7 parsl.dataflow.memoization INFO: Task 1338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b736a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1338 try 0 launched on executor multi with executor id 1339 parsl.dataflow.dflow INFO: Standard output for task 1338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/19de03a4-f313-4a7e-8535-b8e357a13763_isrForCrosstalkSources_1125940_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/19de03a4-f313-4a7e-8535-b8e357a13763_isrForCrosstalkSources_1125940_34.stderr parsl.dataflow.dflow DEBUG: Task 1339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1339 has memoization hash 25542a0b517937cb77bea874212b5bbe parsl.dataflow.memoization INFO: Task 1339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b734c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1339 try 0 launched on executor multi with executor id 1340 parsl.dataflow.dflow INFO: Standard output for task 1339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/64b8f0ad-25dc-4c3d-ba57-7d658b1b2958_isrForCrosstalkSources_1125919_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/64b8f0ad-25dc-4c3d-ba57-7d658b1b2958_isrForCrosstalkSources_1125919_58.stderr parsl.dataflow.dflow DEBUG: Task 1340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1340 has memoization hash d233b85bb9bde7a534cd87c06f102f3a parsl.dataflow.memoization INFO: Task 1340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1340 try 0 launched on executor multi with executor id 1341 parsl.dataflow.dflow INFO: Standard output for task 1340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/148b07a1-ffac-4cb9-bd83-aa98ea23c58d_isrForCrosstalkSources_1125914_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/148b07a1-ffac-4cb9-bd83-aa98ea23c58d_isrForCrosstalkSources_1125914_17.stderr parsl.dataflow.dflow DEBUG: Task 1341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1341 has memoization hash ff5db0b3bf6318c11139dc25ef75a6d3 parsl.dataflow.memoization INFO: Task 1341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b722a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1341 try 0 launched on executor multi with executor id 1342 parsl.dataflow.dflow INFO: Standard output for task 1341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/f4182646-63ca-44af-9278-5ae7082585a8_isrForCrosstalkSources_1125925_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/f4182646-63ca-44af-9278-5ae7082585a8_isrForCrosstalkSources_1125925_13.stderr parsl.dataflow.dflow DEBUG: Task 1342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1342 has memoization hash 31d2fa39e9006e7ac73fa0605ee2bc4f parsl.dataflow.memoization INFO: Task 1342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1342 try 0 launched on executor multi with executor id 1343 parsl.dataflow.dflow INFO: Standard output for task 1342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/110cd67b-5008-4c26-93ba-42d165adea2a_isrForCrosstalkSources_1125912_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/110cd67b-5008-4c26-93ba-42d165adea2a_isrForCrosstalkSources_1125912_55.stderr parsl.dataflow.dflow DEBUG: Task 1343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1343 has memoization hash 602c885d2bb6788e926d3b66a9289de2 parsl.dataflow.memoization INFO: Task 1343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1343 try 0 launched on executor multi with executor id 1344 parsl.dataflow.dflow INFO: Standard output for task 1343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/46259f0f-a2db-4255-b326-b466a232f8a3_isrForCrosstalkSources_1125942_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/46259f0f-a2db-4255-b326-b466a232f8a3_isrForCrosstalkSources_1125942_9.stderr parsl.dataflow.dflow DEBUG: Task 1344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1344 has memoization hash 65720b8695c6f5c99c3d6841d4aa8875 parsl.dataflow.memoization INFO: Task 1344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1344 try 0 launched on executor multi with executor id 1345 parsl.dataflow.dflow INFO: Standard output for task 1344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/e8930649-0e10-4ff8-a775-a34741b0ab65_isrForCrosstalkSources_1125898_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/e8930649-0e10-4ff8-a775-a34741b0ab65_isrForCrosstalkSources_1125898_55.stderr parsl.dataflow.dflow DEBUG: Task 1345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1345 has memoization hash 3bd5b0d9225d46606eecc8f19dc207da parsl.dataflow.memoization INFO: Task 1345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b702c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1345 try 0 launched on executor multi with executor id 1346 parsl.dataflow.dflow INFO: Standard output for task 1345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/a5daaa78-ce74-4ca7-8175-0a80a3f46602_isrForCrosstalkSources_1125907_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/a5daaa78-ce74-4ca7-8175-0a80a3f46602_isrForCrosstalkSources_1125907_24.stderr parsl.dataflow.dflow DEBUG: Task 1346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1346 has memoization hash 10f7132a985eddc5f34ea5c8dceb68e9 parsl.dataflow.memoization INFO: Task 1346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b700e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1346 try 0 launched on executor multi with executor id 1347 parsl.dataflow.dflow INFO: Standard output for task 1346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/4bd815d1-d280-4cff-922e-2de196b9e35f_isrForCrosstalkSources_1125939_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/4bd815d1-d280-4cff-922e-2de196b9e35f_isrForCrosstalkSources_1125939_54.stderr parsl.dataflow.dflow DEBUG: Task 1347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1347 has memoization hash 277e4335768addc1a7868390726ef6f1 parsl.dataflow.memoization INFO: Task 1347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b704a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1347 try 0 launched on executor multi with executor id 1348 parsl.dataflow.dflow INFO: Standard output for task 1347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/58be2540-7f15-4b20-8e6f-d65eeb9ed255_isrForCrosstalkSources_1125940_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/58be2540-7f15-4b20-8e6f-d65eeb9ed255_isrForCrosstalkSources_1125940_40.stderr parsl.dataflow.dflow DEBUG: Task 1348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1348 has memoization hash 3e0c5e095badd090a469583fee356b68 parsl.dataflow.memoization INFO: Task 1348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b705e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1348 try 0 launched on executor multi with executor id 1349 parsl.dataflow.dflow INFO: Standard output for task 1348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/5c49a2cf-3a7e-48cf-af6c-b8ae7c7424df_isrForCrosstalkSources_1125927_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/5c49a2cf-3a7e-48cf-af6c-b8ae7c7424df_isrForCrosstalkSources_1125927_14.stderr parsl.dataflow.dflow DEBUG: Task 1349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1349 has memoization hash 54726f6e840d2888a7def361978c7616 parsl.dataflow.memoization INFO: Task 1349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1349 try 0 launched on executor multi with executor id 1350 parsl.dataflow.dflow INFO: Standard output for task 1349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/8907f801-e875-4a49-b80d-55c21d96d905_isrForCrosstalkSources_1125895_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/8907f801-e875-4a49-b80d-55c21d96d905_isrForCrosstalkSources_1125895_1.stderr parsl.dataflow.dflow DEBUG: Task 1350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1350 has memoization hash b363289e807d29ca857f7273c6adb1b2 parsl.dataflow.memoization INFO: Task 1350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1350 try 0 launched on executor multi with executor id 1351 parsl.dataflow.dflow INFO: Standard output for task 1350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e5801472-6963-40df-b8b0-6158b1cbe440_isrForCrosstalkSources_1125895_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e5801472-6963-40df-b8b0-6158b1cbe440_isrForCrosstalkSources_1125895_46.stderr parsl.dataflow.dflow DEBUG: Task 1351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1351 has memoization hash 7ac18d9c04a45277b86ae3ae86eb1f36 parsl.dataflow.memoization INFO: Task 1351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b707c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1351 try 0 launched on executor multi with executor id 1352 parsl.dataflow.dflow INFO: Standard output for task 1351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/26fe864f-4b6d-4d6b-826e-b1f9d6a22012_isrForCrosstalkSources_1125940_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/26fe864f-4b6d-4d6b-826e-b1f9d6a22012_isrForCrosstalkSources_1125940_51.stderr parsl.dataflow.dflow DEBUG: Task 1352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1352 has memoization hash b9d02da726f96b997f62357c56e02849 parsl.dataflow.memoization INFO: Task 1352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1352 try 0 launched on executor multi with executor id 1353 parsl.dataflow.dflow INFO: Standard output for task 1352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/e775f15e-5882-4b4b-96a0-013d3528f0e4_isrForCrosstalkSources_1125910_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/e775f15e-5882-4b4b-96a0-013d3528f0e4_isrForCrosstalkSources_1125910_41.stderr parsl.dataflow.dflow DEBUG: Task 1353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1353 has memoization hash fe6c58e6ef4c0e4b066af32c678dd5b6 parsl.dataflow.memoization INFO: Task 1353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1353 try 0 launched on executor multi with executor id 1354 parsl.dataflow.dflow INFO: Standard output for task 1353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/1a44c668-988d-40fa-866b-27b986eca3a1_isrForCrosstalkSources_1125893_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/1a44c668-988d-40fa-866b-27b986eca3a1_isrForCrosstalkSources_1125893_15.stderr parsl.dataflow.dflow DEBUG: Task 1354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1354 has memoization hash 5f9941fcb25a268ed4d4795fe64f6100 parsl.dataflow.memoization INFO: Task 1354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1354 try 0 launched on executor multi with executor id 1355 parsl.dataflow.dflow INFO: Standard output for task 1354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/a5f3129d-f7a4-4981-8e45-0e9e96e91f49_isrForCrosstalkSources_1125919_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/a5f3129d-f7a4-4981-8e45-0e9e96e91f49_isrForCrosstalkSources_1125919_55.stderr parsl.dataflow.dflow DEBUG: Task 1355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1355 has memoization hash be26339edd73041bed0b63dfe3e346c2 parsl.dataflow.memoization INFO: Task 1355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1355 try 0 launched on executor multi with executor id 1356 parsl.dataflow.dflow INFO: Standard output for task 1355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/4d872d71-36bd-4eb5-a686-d4e2e5e395c0_isrForCrosstalkSources_1125921_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/4d872d71-36bd-4eb5-a686-d4e2e5e395c0_isrForCrosstalkSources_1125921_39.stderr parsl.dataflow.dflow DEBUG: Task 1356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1356 has memoization hash e25956256174886226c78ce0df5b9b38 parsl.dataflow.memoization INFO: Task 1356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1356 try 0 launched on executor multi with executor id 1357 parsl.dataflow.dflow INFO: Standard output for task 1356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/eeb1eaac-34fb-4809-877c-9766f4082b02_isrForCrosstalkSources_1125893_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/eeb1eaac-34fb-4809-877c-9766f4082b02_isrForCrosstalkSources_1125893_18.stderr parsl.dataflow.dflow DEBUG: Task 1357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1357 has memoization hash e3410c9fcc96c13b8b5b32b82789f240 parsl.dataflow.memoization INFO: Task 1357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1357 try 0 launched on executor multi with executor id 1358 parsl.dataflow.dflow INFO: Standard output for task 1357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/5f3e1adc-353b-411f-aa30-7b7e2060e275_isrForCrosstalkSources_1125940_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/5f3e1adc-353b-411f-aa30-7b7e2060e275_isrForCrosstalkSources_1125940_49.stderr parsl.dataflow.dflow DEBUG: Task 1358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1358 has memoization hash 0399fcb56c2944f6155e0e57b40d1104 parsl.dataflow.memoization INFO: Task 1358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1358 try 0 launched on executor multi with executor id 1359 parsl.dataflow.dflow INFO: Standard output for task 1358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/3e1bb95d-908a-4f48-a68b-5ce0a4e53137_isrForCrosstalkSources_1125901_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/3e1bb95d-908a-4f48-a68b-5ce0a4e53137_isrForCrosstalkSources_1125901_8.stderr parsl.dataflow.dflow DEBUG: Task 1359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1359 has memoization hash 35794e3a81fe8aae8cdb32a522ee1bc6 parsl.dataflow.memoization INFO: Task 1359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b709a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1359 try 0 launched on executor multi with executor id 1360 parsl.dataflow.dflow INFO: Standard output for task 1359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/01bce10b-6b78-4985-a874-fe0f84f75f5f_isrForCrosstalkSources_1125914_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/01bce10b-6b78-4985-a874-fe0f84f75f5f_isrForCrosstalkSources_1125914_16.stderr parsl.dataflow.dflow DEBUG: Task 1360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1360 has memoization hash 5fc15bcb56b5c98fcb4dcc1f87b4c19e parsl.dataflow.memoization INFO: Task 1360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:48 proc_lsst.multi:170 [INFO] [multi] job_id 16896409 proc_lsst.multi INFO: [multi] job_id 16896409 2024-03-07 15:06:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 15:06:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896409 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896409 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16896409 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-07 15:06:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-07 15:06:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 15:06:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1360 try 0 launched on executor multi with executor id 1361 parsl.dataflow.dflow INFO: Standard output for task 1360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/a0382ee6-9d22-4734-af4e-f39768ba91c0_isrForCrosstalkSources_1125930_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/a0382ee6-9d22-4734-af4e-f39768ba91c0_isrForCrosstalkSources_1125930_17.stderr parsl.dataflow.dflow DEBUG: Task 1361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1361 has memoization hash a1c7cb7e3a9707b7618f5a9817eee2c0 parsl.dataflow.memoization INFO: Task 1361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1361 try 0 launched on executor multi with executor id 1362 parsl.dataflow.dflow INFO: Standard output for task 1361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/764870d6-0313-494d-8c3b-5f3dbbdca103_isrForCrosstalkSources_1125914_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/764870d6-0313-494d-8c3b-5f3dbbdca103_isrForCrosstalkSources_1125914_27.stderr parsl.dataflow.dflow DEBUG: Task 1362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1362 has memoization hash 613fa2d6617c03f65dff215c1a14b5a5 parsl.dataflow.memoization INFO: Task 1362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1362 try 0 launched on executor multi with executor id 1363 parsl.dataflow.dflow INFO: Standard output for task 1362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/bf4c0c24-b984-48c2-9874-2456c249e54a_isrForCrosstalkSources_1125944_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/bf4c0c24-b984-48c2-9874-2456c249e54a_isrForCrosstalkSources_1125944_54.stderr parsl.dataflow.dflow DEBUG: Task 1363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1363 has memoization hash e09c5d8f3510c0269ec49f99a92bc2d7 parsl.dataflow.memoization INFO: Task 1363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1363 try 0 launched on executor multi with executor id 1364 parsl.dataflow.dflow INFO: Standard output for task 1363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/5e5af668-3765-4a46-a51a-85fda1b5d943_isrForCrosstalkSources_1125903_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/5e5af668-3765-4a46-a51a-85fda1b5d943_isrForCrosstalkSources_1125903_41.stderr parsl.dataflow.dflow DEBUG: Task 1364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1364 has memoization hash fe36b12ab89ef769e909ca1ba4c35f66 parsl.dataflow.memoization INFO: Task 1364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1364 try 0 launched on executor multi with executor id 1365 parsl.dataflow.dflow INFO: Standard output for task 1364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/01d360e9-a48a-464e-9c79-05634e49d91b_isrForCrosstalkSources_1125923_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/01d360e9-a48a-464e-9c79-05634e49d91b_isrForCrosstalkSources_1125923_35.stderr parsl.dataflow.dflow DEBUG: Task 1365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1365 has memoization hash 5db82598814d9b848ca4bc01d571e950 parsl.dataflow.memoization INFO: Task 1365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b720c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1365 try 0 launched on executor multi with executor id 1366 parsl.dataflow.dflow INFO: Standard output for task 1365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/93764c76-9d90-4375-ba55-a06edef6cfbd_isrForCrosstalkSources_1125937_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/93764c76-9d90-4375-ba55-a06edef6cfbd_isrForCrosstalkSources_1125937_19.stderr parsl.dataflow.dflow DEBUG: Task 1366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1366 has memoization hash 9db4202212cfebad38fff18c625b49c0 parsl.dataflow.memoization INFO: Task 1366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1366 try 0 launched on executor multi with executor id 1367 parsl.dataflow.dflow INFO: Standard output for task 1366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/4f7fd9d4-f7d1-4f46-946a-4db13e423148_isrForCrosstalkSources_1125947_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/4f7fd9d4-f7d1-4f46-946a-4db13e423148_isrForCrosstalkSources_1125947_43.stderr parsl.dataflow.dflow DEBUG: Task 1367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1367 has memoization hash 2e9e6611ea27a5258576fae0ad1424cd parsl.dataflow.memoization INFO: Task 1367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1367 try 0 launched on executor multi with executor id 1368 parsl.dataflow.dflow INFO: Standard output for task 1367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/eabec1bc-566e-45c6-9af7-a474bbe49d24_isrForCrosstalkSources_1125919_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/eabec1bc-566e-45c6-9af7-a474bbe49d24_isrForCrosstalkSources_1125919_27.stderr parsl.dataflow.dflow DEBUG: Task 1368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1368 has memoization hash ffc142dcb608997c8d9f95f98710cc67 parsl.dataflow.memoization INFO: Task 1368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1368 try 0 launched on executor multi with executor id 1369 parsl.dataflow.dflow INFO: Standard output for task 1368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/65c8c05e-bc34-4a81-9ecf-ef6f44d6f76e_isrForCrosstalkSources_1125912_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/65c8c05e-bc34-4a81-9ecf-ef6f44d6f76e_isrForCrosstalkSources_1125912_26.stderr parsl.dataflow.dflow DEBUG: Task 1369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1369 has memoization hash 0fcd80f1fbe62c3080c3247ddebad944 parsl.dataflow.memoization INFO: Task 1369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1369 try 0 launched on executor multi with executor id 1370 parsl.dataflow.dflow INFO: Standard output for task 1369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d73ec0d6-0925-4a60-b130-75f7494cf85a_isrForCrosstalkSources_1125890_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d73ec0d6-0925-4a60-b130-75f7494cf85a_isrForCrosstalkSources_1125890_16.stderr parsl.dataflow.dflow DEBUG: Task 1370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1370 has memoization hash ba35095a9e15e440f051b35bb2ac5461 parsl.dataflow.memoization INFO: Task 1370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1370 try 0 launched on executor multi with executor id 1371 parsl.dataflow.dflow INFO: Standard output for task 1370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/dac9dc43-dcce-49b2-9c12-1213156e7dc6_isrForCrosstalkSources_1125913_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/dac9dc43-dcce-49b2-9c12-1213156e7dc6_isrForCrosstalkSources_1125913_13.stderr parsl.dataflow.dflow DEBUG: Task 1371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1371 has memoization hash a85603243d7e18d7e3eb09cc83bdeaf1 parsl.dataflow.memoization INFO: Task 1371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1371 try 0 launched on executor multi with executor id 1372 parsl.dataflow.dflow INFO: Standard output for task 1371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/a686a204-7a05-4614-b870-55034ea05ac8_isrForCrosstalkSources_1125891_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/a686a204-7a05-4614-b870-55034ea05ac8_isrForCrosstalkSources_1125891_41.stderr parsl.dataflow.dflow DEBUG: Task 1372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1372 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1372 has memoization hash 5c9c8d71f1ac9e804d62c353e5f71bb9 parsl.dataflow.memoization INFO: Task 1372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1372 try 0 launched on executor multi with executor id 1373 parsl.dataflow.dflow INFO: Standard output for task 1372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/2ff8ffb7-0ab3-41fe-9504-ab3ffe235b99_isrForCrosstalkSources_1125937_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/2ff8ffb7-0ab3-41fe-9504-ab3ffe235b99_isrForCrosstalkSources_1125937_24.stderr parsl.dataflow.dflow DEBUG: Task 1373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1373 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1373 has memoization hash 441450609ce89c3f1a674fced532bb7f parsl.dataflow.memoization INFO: Task 1373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1373 try 0 launched on executor multi with executor id 1374 parsl.dataflow.dflow INFO: Standard output for task 1373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/342a14f6-c88a-4d60-be71-66c6c7b18088_isrForCrosstalkSources_1125899_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/342a14f6-c88a-4d60-be71-66c6c7b18088_isrForCrosstalkSources_1125899_1.stderr parsl.dataflow.dflow DEBUG: Task 1374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1374 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1374 has memoization hash 24900b6c71a330b98ebf6295e3e50cea parsl.dataflow.memoization INFO: Task 1374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1374 try 0 launched on executor multi with executor id 1375 parsl.dataflow.dflow INFO: Standard output for task 1374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/61205585-ca3c-448b-bfed-5f6ed8ef2727_isrForCrosstalkSources_1125900_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/61205585-ca3c-448b-bfed-5f6ed8ef2727_isrForCrosstalkSources_1125900_55.stderr parsl.dataflow.dflow DEBUG: Task 1375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1375 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1375 has memoization hash 8587e28859857ad58614e80d4a5402ff parsl.dataflow.memoization INFO: Task 1375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1375 try 0 launched on executor multi with executor id 1376 parsl.dataflow.dflow INFO: Standard output for task 1375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/291e20e4-ee50-48be-9bea-53b97df8659f_isrForCrosstalkSources_1125920_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/291e20e4-ee50-48be-9bea-53b97df8659f_isrForCrosstalkSources_1125920_56.stderr parsl.dataflow.dflow DEBUG: Task 1376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1376 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1376 has memoization hash ddb4a4418522d850a136260dab2f0b07 parsl.dataflow.memoization INFO: Task 1376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b718a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1376 try 0 launched on executor multi with executor id 1377 parsl.dataflow.dflow INFO: Standard output for task 1376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/55379f22-c4bb-4334-a727-52538802d1e7_isrForCrosstalkSources_1125895_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/55379f22-c4bb-4334-a727-52538802d1e7_isrForCrosstalkSources_1125895_41.stderr parsl.dataflow.dflow DEBUG: Task 1377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1377 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1377 has memoization hash 14a791315b65319a64e8cea444488fc2 parsl.dataflow.memoization INFO: Task 1377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b719e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1377 try 0 launched on executor multi with executor id 1378 parsl.dataflow.dflow INFO: Standard output for task 1377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/0245f702-bb85-4234-84e7-93612e7cf9fe_isrForCrosstalkSources_1125925_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/0245f702-bb85-4234-84e7-93612e7cf9fe_isrForCrosstalkSources_1125925_62.stderr parsl.dataflow.dflow DEBUG: Task 1378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1378 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1378 has memoization hash b1c683c923606bd0320960bdf10addef parsl.dataflow.memoization INFO: Task 1378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1378 try 0 launched on executor multi with executor id 1379 parsl.dataflow.dflow INFO: Standard output for task 1378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/aa34c9b0-ba58-4c80-a760-a1f0b57f8774_isrForCrosstalkSources_1125931_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/aa34c9b0-ba58-4c80-a760-a1f0b57f8774_isrForCrosstalkSources_1125931_45.stderr parsl.dataflow.dflow DEBUG: Task 1379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1379 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1379 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1379 has memoization hash fe63abebec5bee86245db2bdd7a91184 parsl.dataflow.memoization INFO: Task 1379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1379 try 0 launched on executor multi with executor id 1380 parsl.dataflow.dflow INFO: Standard output for task 1379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/ce8f83ce-367f-4507-b98a-cbd602570333_isrForCrosstalkSources_1125929_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/ce8f83ce-367f-4507-b98a-cbd602570333_isrForCrosstalkSources_1125929_21.stderr parsl.dataflow.dflow DEBUG: Task 1380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1380 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1380 has memoization hash 0f95c371db5338056336c070e76ed1ce parsl.dataflow.memoization INFO: Task 1380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b716c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1380 try 0 launched on executor multi with executor id 1381 parsl.dataflow.dflow INFO: Standard output for task 1380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/10aa9fc9-a8a6-4c8d-8370-25151756a733_isrForCrosstalkSources_1125909_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/10aa9fc9-a8a6-4c8d-8370-25151756a733_isrForCrosstalkSources_1125909_22.stderr parsl.dataflow.dflow DEBUG: Task 1381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1381 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1381 has memoization hash f476bcc7d8e259fe4430e74695f02abd parsl.dataflow.memoization INFO: Task 1381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1381 try 0 launched on executor multi with executor id 1382 parsl.dataflow.dflow INFO: Standard output for task 1381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/79f214bc-c848-4b9d-9fa6-684aa1d42e9c_isrForCrosstalkSources_1125924_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/79f214bc-c848-4b9d-9fa6-684aa1d42e9c_isrForCrosstalkSources_1125924_2.stderr parsl.dataflow.dflow DEBUG: Task 1382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1382 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1382 has memoization hash 9dc395325ad0f9397bb846d3047f9373 parsl.dataflow.memoization INFO: Task 1382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b714e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1382 try 0 launched on executor multi with executor id 1383 parsl.dataflow.dflow INFO: Standard output for task 1382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/9ac60577-87cf-43c3-a8fb-ad098dc88d64_isrForCrosstalkSources_1125917_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/9ac60577-87cf-43c3-a8fb-ad098dc88d64_isrForCrosstalkSources_1125917_38.stderr parsl.dataflow.dflow DEBUG: Task 1383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1383 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1383 has memoization hash e13d170adcebfd01f639dda8015a3cac parsl.dataflow.memoization INFO: Task 1383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1383 try 0 launched on executor multi with executor id 1384 parsl.dataflow.dflow INFO: Standard output for task 1383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/667a6410-f3fe-4cff-ab0c-92372e874397_isrForCrosstalkSources_1125911_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/667a6410-f3fe-4cff-ab0c-92372e874397_isrForCrosstalkSources_1125911_51.stderr parsl.dataflow.dflow DEBUG: Task 1384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1384 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1384 has memoization hash e2af815750afe13eb246551b514cb328 parsl.dataflow.memoization INFO: Task 1384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:49 proc_lsst.multi:170 [INFO] [multi] job_id 16896410 proc_lsst.multi INFO: [multi] job_id 16896410 2024-03-07 15:06:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 15:06:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896410 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896410 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16896410 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-07 15:06:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-07 15:06:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 15:06:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1384 try 0 launched on executor multi with executor id 1385 parsl.dataflow.dflow INFO: Standard output for task 1384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/5e8d9d7f-41a7-4f50-b9c8-fe1e07375114_isrForCrosstalkSources_1125905_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/5e8d9d7f-41a7-4f50-b9c8-fe1e07375114_isrForCrosstalkSources_1125905_51.stderr parsl.dataflow.dflow DEBUG: Task 1385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1385 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1385 has memoization hash 7d82c253615fcd45c43fcc8c001a2acd parsl.dataflow.memoization INFO: Task 1385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1385 try 0 launched on executor multi with executor id 1386 parsl.dataflow.dflow INFO: Standard output for task 1385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e8aa9e70-814b-4737-9f32-8ba08f9831bd_isrForCrosstalkSources_1125937_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e8aa9e70-814b-4737-9f32-8ba08f9831bd_isrForCrosstalkSources_1125937_9.stderr parsl.dataflow.dflow DEBUG: Task 1386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1386 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1386 has memoization hash 5c2e62eb117924e96d75a467074602fd parsl.dataflow.memoization INFO: Task 1386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1386 try 0 launched on executor multi with executor id 1387 parsl.dataflow.dflow INFO: Standard output for task 1386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/384eb45d-7d50-419f-a499-1fc4c1ecdeab_isrForCrosstalkSources_1125904_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/384eb45d-7d50-419f-a499-1fc4c1ecdeab_isrForCrosstalkSources_1125904_58.stderr parsl.dataflow.dflow DEBUG: Task 1387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1387 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1387 has memoization hash 04d9b1f270d53895524c4a2be6da693c parsl.dataflow.memoization INFO: Task 1387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b711c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1387 try 0 launched on executor multi with executor id 1388 parsl.dataflow.dflow INFO: Standard output for task 1387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/bd0a4bb1-6e25-479f-9d66-dab1c40e8e8a_isrForCrosstalkSources_1125915_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/bd0a4bb1-6e25-479f-9d66-dab1c40e8e8a_isrForCrosstalkSources_1125915_20.stderr parsl.dataflow.dflow DEBUG: Task 1388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1388 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1388 has memoization hash 0772f88234ffd27339446b380057b1c8 parsl.dataflow.memoization INFO: Task 1388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b713a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1388 try 0 launched on executor multi with executor id 1389 parsl.dataflow.dflow INFO: Standard output for task 1388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/adb1e325-f23b-45f5-a863-3c6352ec9be0_isrForCrosstalkSources_1125914_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/adb1e325-f23b-45f5-a863-3c6352ec9be0_isrForCrosstalkSources_1125914_57.stderr parsl.dataflow.dflow DEBUG: Task 1389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1389 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1389 has memoization hash 536c68c3d6f9834e929d59a1114822a6 parsl.dataflow.memoization INFO: Task 1389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1389 try 0 launched on executor multi with executor id 1390 parsl.dataflow.dflow INFO: Standard output for task 1389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/073f7a94-81c6-4c0c-9759-e987b478ab57_isrForCrosstalkSources_1125892_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/073f7a94-81c6-4c0c-9759-e987b478ab57_isrForCrosstalkSources_1125892_10.stderr parsl.dataflow.dflow DEBUG: Task 1390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1390 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1390 has memoization hash 084b654e792ae5de6251d463afbbae48 parsl.dataflow.memoization INFO: Task 1390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1390 try 0 launched on executor multi with executor id 1391 parsl.dataflow.dflow INFO: Standard output for task 1390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a96232ed-23de-4bdb-a4e5-784b5a1dccb7_isrForCrosstalkSources_1125938_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a96232ed-23de-4bdb-a4e5-784b5a1dccb7_isrForCrosstalkSources_1125938_60.stderr parsl.dataflow.dflow DEBUG: Task 1391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1391 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1391 has memoization hash 869882de80b44cd6afbfe4158566aa3e parsl.dataflow.memoization INFO: Task 1391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1391 try 0 launched on executor multi with executor id 1392 parsl.dataflow.dflow INFO: Standard output for task 1391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/f2be6914-59dc-4747-b9a8-a7f3a1a55145_isrForCrosstalkSources_1125909_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/f2be6914-59dc-4747-b9a8-a7f3a1a55145_isrForCrosstalkSources_1125909_57.stderr parsl.dataflow.dflow DEBUG: Task 1392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1392 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1392 has memoization hash 992d5f3f7dddc2169dc2a81bb8f6bd34 parsl.dataflow.memoization INFO: Task 1392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1392 try 0 launched on executor multi with executor id 1393 parsl.dataflow.dflow INFO: Standard output for task 1392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/a993ea55-174f-4c06-a240-e8d2c508ab66_isrForCrosstalkSources_1125890_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/a993ea55-174f-4c06-a240-e8d2c508ab66_isrForCrosstalkSources_1125890_39.stderr parsl.dataflow.dflow DEBUG: Task 1393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1393 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1393 has memoization hash 994eda556e0473f0accff5f5ebf67af0 parsl.dataflow.memoization INFO: Task 1393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1393 try 0 launched on executor multi with executor id 1394 parsl.dataflow.dflow INFO: Standard output for task 1393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/00418d17-30f5-4ac1-a691-cdeccf6815b9_isrForCrosstalkSources_1125908_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/00418d17-30f5-4ac1-a691-cdeccf6815b9_isrForCrosstalkSources_1125908_20.stderr parsl.dataflow.dflow DEBUG: Task 1394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1394 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1394 has memoization hash 04afdfa82b8af72379644f1650bc9394 parsl.dataflow.memoization INFO: Task 1394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1394 try 0 launched on executor multi with executor id 1395 parsl.dataflow.dflow INFO: Standard output for task 1394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4ab5a2e9-3541-451e-be99-e6d632d882c2_isrForCrosstalkSources_1125908_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4ab5a2e9-3541-451e-be99-e6d632d882c2_isrForCrosstalkSources_1125908_3.stderr parsl.dataflow.dflow DEBUG: Task 1395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1395 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1395 has memoization hash dc648a069617290cca0b13ebe3875a51 parsl.dataflow.memoization INFO: Task 1395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1395 try 0 launched on executor multi with executor id 1396 parsl.dataflow.dflow INFO: Standard output for task 1395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/cd88cf6a-18cc-4158-bc0c-2d4a95c280d7_isrForCrosstalkSources_1125931_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/cd88cf6a-18cc-4158-bc0c-2d4a95c280d7_isrForCrosstalkSources_1125931_62.stderr parsl.dataflow.dflow DEBUG: Task 1396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1396 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1396 has memoization hash 631c67fed881db80c4e5893b0003d34c parsl.dataflow.memoization INFO: Task 1396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1396 try 0 launched on executor multi with executor id 1397 parsl.dataflow.dflow INFO: Standard output for task 1396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/c9295d97-0a61-4381-8f26-f70f2891ef93_isrForCrosstalkSources_1125919_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/c9295d97-0a61-4381-8f26-f70f2891ef93_isrForCrosstalkSources_1125919_39.stderr parsl.dataflow.dflow DEBUG: Task 1397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1397 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1397 has memoization hash 575e9f7869c66475855467ce175b2276 parsl.dataflow.memoization INFO: Task 1397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b732e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1397 try 0 launched on executor multi with executor id 1398 parsl.dataflow.dflow INFO: Standard output for task 1397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/e4b8cb45-2295-469d-82ef-bc8bae9cb6ae_isrForCrosstalkSources_1125920_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/e4b8cb45-2295-469d-82ef-bc8bae9cb6ae_isrForCrosstalkSources_1125920_58.stderr parsl.dataflow.dflow DEBUG: Task 1398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1398 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1398 has memoization hash 058b7b6a0979f040104a0f7d7003ff40 parsl.dataflow.memoization INFO: Task 1398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1398 try 0 launched on executor multi with executor id 1399 parsl.dataflow.dflow INFO: Standard output for task 1398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/e8702108-2978-4a1d-9206-d5664c5dbab0_isrForCrosstalkSources_1125896_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/e8702108-2978-4a1d-9206-d5664c5dbab0_isrForCrosstalkSources_1125896_57.stderr parsl.dataflow.dflow DEBUG: Task 1399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1399 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1399 has memoization hash 343021e59f5defa98629aed08d9560a5 parsl.dataflow.memoization INFO: Task 1399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b731a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1399 try 0 launched on executor multi with executor id 1400 parsl.dataflow.dflow INFO: Standard output for task 1399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/1b884800-95ea-4c4f-ad1d-5586c02b0953_isrForCrosstalkSources_1125920_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/1b884800-95ea-4c4f-ad1d-5586c02b0953_isrForCrosstalkSources_1125920_48.stderr parsl.dataflow.dflow DEBUG: Task 1400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1400 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1400 has memoization hash a4fe675b36971caee45403aff40172f6 parsl.dataflow.memoization INFO: Task 1400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b71620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1400 try 0 launched on executor multi with executor id 1401 parsl.dataflow.dflow INFO: Standard output for task 1400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/ba49327c-4e66-4094-a384-375d36f9470a_isrForCrosstalkSources_1125892_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/ba49327c-4e66-4094-a384-375d36f9470a_isrForCrosstalkSources_1125892_15.stderr parsl.dataflow.dflow DEBUG: Task 1401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1401 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1401 has memoization hash d08fe4e6853655d2fc2dfb6e0ea5b545 parsl.dataflow.memoization INFO: Task 1401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1401 try 0 launched on executor multi with executor id 1402 parsl.dataflow.dflow INFO: Standard output for task 1401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/6c717a7d-b0e8-4867-b021-d226d4820886_isrForCrosstalkSources_1125914_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/6c717a7d-b0e8-4867-b021-d226d4820886_isrForCrosstalkSources_1125914_8.stderr parsl.dataflow.dflow DEBUG: Task 1402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1402 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1402 has memoization hash 32a4f8250b4eed20cfdc4b602d792ea2 parsl.dataflow.memoization INFO: Task 1402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1402 try 0 launched on executor multi with executor id 1403 parsl.dataflow.dflow INFO: Standard output for task 1402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/b37d4f84-fb8e-4995-9c20-faec38961f06_isrForCrosstalkSources_1125915_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/b37d4f84-fb8e-4995-9c20-faec38961f06_isrForCrosstalkSources_1125915_39.stderr parsl.dataflow.dflow DEBUG: Task 1403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1403 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1403 has memoization hash 630eaa3a625dde32630332cc3dacfa47 parsl.dataflow.memoization INFO: Task 1403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1403 try 0 launched on executor multi with executor id 1404 parsl.dataflow.dflow INFO: Standard output for task 1403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/53c6e8ad-2305-4cb1-843d-86031797e63f_isrForCrosstalkSources_1125926_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/53c6e8ad-2305-4cb1-843d-86031797e63f_isrForCrosstalkSources_1125926_9.stderr parsl.dataflow.dflow DEBUG: Task 1404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1404 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1404 has memoization hash 98e8119593637c525c51af7aea810933 parsl.dataflow.memoization INFO: Task 1404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1404 try 0 launched on executor multi with executor id 1405 parsl.dataflow.dflow INFO: Standard output for task 1404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/17d75be1-5b39-40f4-a28b-2c8f04cf8438_isrForCrosstalkSources_1125940_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/17d75be1-5b39-40f4-a28b-2c8f04cf8438_isrForCrosstalkSources_1125940_15.stderr parsl.dataflow.dflow DEBUG: Task 1405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1405 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1405 has memoization hash b1943eaa7a217978632e12a026f3d6aa parsl.dataflow.memoization INFO: Task 1405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1405 try 0 launched on executor multi with executor id 1406 parsl.dataflow.dflow INFO: Standard output for task 1405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/1590cbce-d53b-4acb-aec4-8f961a2bf582_isrForCrosstalkSources_1125943_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/1590cbce-d53b-4acb-aec4-8f961a2bf582_isrForCrosstalkSources_1125943_13.stderr parsl.dataflow.dflow DEBUG: Task 1406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1406 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1406 has memoization hash 66260238ec4c847933da1f2f7a7c2f89 parsl.dataflow.memoization INFO: Task 1406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1406 try 0 launched on executor multi with executor id 1407 parsl.dataflow.dflow INFO: Standard output for task 1406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/dd05369b-0577-4ba0-90d3-0c15e0106eaa_isrForCrosstalkSources_1125925_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/dd05369b-0577-4ba0-90d3-0c15e0106eaa_isrForCrosstalkSources_1125925_34.stderr parsl.dataflow.dflow DEBUG: Task 1407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1407 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1407 has memoization hash 1102f1f7eae7509d64d6972abad25c4a parsl.dataflow.memoization INFO: Task 1407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1407 try 0 launched on executor multi with executor id 1408 parsl.dataflow.dflow INFO: Standard output for task 1407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/adbc05b6-b340-43fa-b3c6-31eefdbfcbe6_isrForCrosstalkSources_1125945_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/adbc05b6-b340-43fa-b3c6-31eefdbfcbe6_isrForCrosstalkSources_1125945_31.stderr parsl.dataflow.dflow DEBUG: Task 1408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1408 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1408 has memoization hash 62c0f01d18419428428d8c73d27ff0a4 parsl.dataflow.memoization INFO: Task 1408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1408 try 0 launched on executor multi with executor id 1409 parsl.dataflow.dflow INFO: Standard output for task 1408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/7966bd7e-ee34-46a4-9577-0dcbd70eb6a6_isrForCrosstalkSources_1125894_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/7966bd7e-ee34-46a4-9577-0dcbd70eb6a6_isrForCrosstalkSources_1125894_13.stderr parsl.dataflow.dflow DEBUG: Task 1409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1409 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1409 has memoization hash 894ce4c93bb7180b75d6f13e839994b2 parsl.dataflow.memoization INFO: Task 1409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1409 try 0 launched on executor multi with executor id 1410 parsl.dataflow.dflow INFO: Standard output for task 1409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/fa57b67e-c264-41e7-b7fd-160984be1c75_isrForCrosstalkSources_1125923_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/fa57b67e-c264-41e7-b7fd-160984be1c75_isrForCrosstalkSources_1125923_8.stderr parsl.dataflow.dflow DEBUG: Task 1410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1410 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1410 has memoization hash 676748fd617ab9420879684f6ba13fdd parsl.dataflow.memoization INFO: Task 1410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b728e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1410 try 0 launched on executor multi with executor id 1411 parsl.dataflow.dflow INFO: Standard output for task 1410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d5a24c4e-3227-4743-9b65-44ee305691a2_isrForCrosstalkSources_1125934_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d5a24c4e-3227-4743-9b65-44ee305691a2_isrForCrosstalkSources_1125934_28.stderr parsl.dataflow.dflow DEBUG: Task 1411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1411 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1411 has memoization hash 819f7fac8e7fab6b593692676ac803be parsl.dataflow.memoization INFO: Task 1411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1411 try 0 launched on executor multi with executor id 1412 parsl.dataflow.dflow INFO: Standard output for task 1411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/ae2cdade-6c68-4e67-b1f4-ecc90472388d_isrForCrosstalkSources_1125909_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/ae2cdade-6c68-4e67-b1f4-ecc90472388d_isrForCrosstalkSources_1125909_16.stderr parsl.dataflow.dflow DEBUG: Task 1412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1412 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1412 has memoization hash 33406869dd118f61e4358d55f911cd32 parsl.dataflow.memoization INFO: Task 1412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1412 try 0 launched on executor multi with executor id 1413 parsl.dataflow.dflow INFO: Standard output for task 1412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/c199e52e-d575-4dd0-be99-d40087b33d41_isrForCrosstalkSources_1125906_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/c199e52e-d575-4dd0-be99-d40087b33d41_isrForCrosstalkSources_1125906_36.stderr parsl.dataflow.dflow DEBUG: Task 1413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1413 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1413 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1413 has memoization hash f80f71afd10177e31f3eac50ae818d9c parsl.dataflow.memoization INFO: Task 1413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b727a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1413 try 0 launched on executor multi with executor id 1414 parsl.dataflow.dflow INFO: Standard output for task 1413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/3d5c6889-a934-4b25-a0a3-521fe20552bc_isrForCrosstalkSources_1125913_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/3d5c6889-a934-4b25-a0a3-521fe20552bc_isrForCrosstalkSources_1125913_57.stderr parsl.dataflow.dflow DEBUG: Task 1414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1414 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1414 has memoization hash 812cac4dd90dee9851d8fa41e1cea0fe parsl.dataflow.memoization INFO: Task 1414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1414 try 0 launched on executor multi with executor id 1415 parsl.dataflow.dflow INFO: Standard output for task 1414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/b09137f4-6fcb-417c-aed2-92456a191feb_isrForCrosstalkSources_1125923_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/b09137f4-6fcb-417c-aed2-92456a191feb_isrForCrosstalkSources_1125923_5.stderr parsl.dataflow.dflow DEBUG: Task 1415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1415 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1415 has memoization hash 9571e840150b9af3363a87fa7fe9c2da parsl.dataflow.memoization INFO: Task 1415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b725c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1415 try 0 launched on executor multi with executor id 1416 parsl.dataflow.dflow INFO: Standard output for task 1415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/0254a014-9d47-460d-b4a3-49f183f0bb90_isrForCrosstalkSources_1125937_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/0254a014-9d47-460d-b4a3-49f183f0bb90_isrForCrosstalkSources_1125937_55.stderr parsl.dataflow.dflow DEBUG: Task 1416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1416 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1416 has memoization hash 10e45a9459e801b4134cbb15cf81d638 parsl.dataflow.memoization INFO: Task 1416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1416 try 0 launched on executor multi with executor id 1417 parsl.dataflow.dflow INFO: Standard output for task 1416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/e9f05a7d-2244-49f1-86f0-d2ce1fa49c3a_isrForCrosstalkSources_1125917_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/e9f05a7d-2244-49f1-86f0-d2ce1fa49c3a_isrForCrosstalkSources_1125917_45.stderr parsl.dataflow.dflow DEBUG: Task 1417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1417 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1417 has memoization hash 27e4aadfef8888f1efeece6ede995ff8 parsl.dataflow.memoization INFO: Task 1417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1417 try 0 launched on executor multi with executor id 1418 parsl.dataflow.dflow INFO: Standard output for task 1417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/768251d2-016a-4fa8-9954-792feb58ee9f_isrForCrosstalkSources_1125904_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/768251d2-016a-4fa8-9954-792feb58ee9f_isrForCrosstalkSources_1125904_48.stderr parsl.dataflow.dflow DEBUG: Task 1418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1418 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1418 has memoization hash c01a7be38dd1b587bb4c9a1f15177abb parsl.dataflow.memoization INFO: Task 1418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1418 try 0 launched on executor multi with executor id 1419 parsl.dataflow.dflow INFO: Standard output for task 1418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/4eaa5b11-89ae-4bb9-bb66-ed820b54ae21_isrForCrosstalkSources_1125899_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/4eaa5b11-89ae-4bb9-bb66-ed820b54ae21_isrForCrosstalkSources_1125899_62.stderr parsl.dataflow.dflow DEBUG: Task 1419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1419 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1419 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1419 has memoization hash bf881e1e2eb809f624562bed83e8f14c parsl.dataflow.memoization INFO: Task 1419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1419 try 0 launched on executor multi with executor id 1420 parsl.dataflow.dflow INFO: Standard output for task 1419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/ce9a9547-eaca-41ff-818c-1cccc4719bd4_isrForCrosstalkSources_1125934_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/ce9a9547-eaca-41ff-818c-1cccc4719bd4_isrForCrosstalkSources_1125934_9.stderr parsl.dataflow.dflow DEBUG: Task 1420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1420 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1420 has memoization hash 820bebb2cf67b24f3375bf156b19b9a7 parsl.dataflow.memoization INFO: Task 1420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1420 try 0 launched on executor multi with executor id 1421 parsl.dataflow.dflow INFO: Standard output for task 1420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/e26c76a1-e4b3-40f2-8213-31999581c438_isrForCrosstalkSources_1125940_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/e26c76a1-e4b3-40f2-8213-31999581c438_isrForCrosstalkSources_1125940_1.stderr parsl.dataflow.dflow DEBUG: Task 1421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1421 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1421 has memoization hash d031b159a403c3921b09d0e483c7ae6e parsl.dataflow.memoization INFO: Task 1421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1421 try 0 launched on executor multi with executor id 1422 parsl.dataflow.dflow INFO: Standard output for task 1421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/8c1cfb76-6b49-4c58-acc8-850593f5e91e_isrForCrosstalkSources_1125890_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/8c1cfb76-6b49-4c58-acc8-850593f5e91e_isrForCrosstalkSources_1125890_18.stderr parsl.dataflow.dflow DEBUG: Task 1422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1422 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1422 has memoization hash b68339ad3b053f207f836210627ba7ee parsl.dataflow.memoization INFO: Task 1422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1422 try 0 launched on executor multi with executor id 1423 parsl.dataflow.dflow INFO: Standard output for task 1422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/81331e89-bc55-48a5-8e98-7ab01ecec6bb_isrForCrosstalkSources_1125900_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/81331e89-bc55-48a5-8e98-7ab01ecec6bb_isrForCrosstalkSources_1125900_47.stderr parsl.dataflow.dflow DEBUG: Task 1423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1423 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1423 has memoization hash 5f642ec500c87750fc249970912e09e3 parsl.dataflow.memoization INFO: Task 1423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b739c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1423 try 0 launched on executor multi with executor id 1424 parsl.dataflow.dflow INFO: Standard output for task 1423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/bc21bd57-5589-4807-acb2-30889b1f4b9c_isrForCrosstalkSources_1125924_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/bc21bd57-5589-4807-acb2-30889b1f4b9c_isrForCrosstalkSources_1125924_5.stderr parsl.dataflow.dflow DEBUG: Task 1424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1424 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1424 has memoization hash 08d0c4fa2d80eeea2362d49cdebb6fa0 parsl.dataflow.memoization INFO: Task 1424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1424 try 0 launched on executor multi with executor id 1425 parsl.dataflow.dflow INFO: Standard output for task 1424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/935657c2-4341-4944-b62f-511d713da1c0_isrForCrosstalkSources_1125933_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/935657c2-4341-4944-b62f-511d713da1c0_isrForCrosstalkSources_1125933_52.stderr parsl.dataflow.dflow DEBUG: Task 1425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1425 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1425 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1425 has memoization hash 6933b16658e1c6869a1e11a0bde7ce82 parsl.dataflow.memoization INFO: Task 1425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1425 try 0 launched on executor multi with executor id 1426 parsl.dataflow.dflow INFO: Standard output for task 1425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/97798546-2d5f-4020-85bb-5eea4bce8ae1_isrForCrosstalkSources_1125909_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/97798546-2d5f-4020-85bb-5eea4bce8ae1_isrForCrosstalkSources_1125909_58.stderr parsl.dataflow.dflow DEBUG: Task 1426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1426 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1426 has memoization hash 7197a2c60c9e86ed1deb9b1ba4afa39b parsl.dataflow.memoization INFO: Task 1426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1426 try 0 launched on executor multi with executor id 1427 parsl.dataflow.dflow INFO: Standard output for task 1426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/430ae6e6-b786-49a0-9378-1c8933bd3ef5_isrForCrosstalkSources_1125939_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/430ae6e6-b786-49a0-9378-1c8933bd3ef5_isrForCrosstalkSources_1125939_58.stderr parsl.dataflow.dflow DEBUG: Task 1427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1427 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1427 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1427 has memoization hash 9887b7a06b11b18ba707b24c8f43e014 parsl.dataflow.memoization INFO: Task 1427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1427 try 0 launched on executor multi with executor id 1428 parsl.dataflow.dflow INFO: Standard output for task 1427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/4a38cf8c-b920-4dce-9971-899838765d89_isrForCrosstalkSources_1125909_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/4a38cf8c-b920-4dce-9971-899838765d89_isrForCrosstalkSources_1125909_36.stderr parsl.dataflow.dflow DEBUG: Task 1428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1428 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1428 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1428 has memoization hash ff2946bb2152dd0d97d12550787df70f parsl.dataflow.memoization INFO: Task 1428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b737e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1428 try 0 launched on executor multi with executor id 1429 parsl.dataflow.dflow INFO: Standard output for task 1428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/3398de58-64f8-4f15-8d92-025a78a21357_isrForCrosstalkSources_1125917_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/3398de58-64f8-4f15-8d92-025a78a21357_isrForCrosstalkSources_1125917_24.stderr parsl.dataflow.dflow DEBUG: Task 1429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1429 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1429 has memoization hash 389a38376d84658d9f48e593eaafca60 parsl.dataflow.memoization INFO: Task 1429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1429 try 0 launched on executor multi with executor id 1430 parsl.dataflow.dflow INFO: Standard output for task 1429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/f12fb531-d5cd-4926-9bd7-fae37d6542a0_isrForCrosstalkSources_1125936_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/f12fb531-d5cd-4926-9bd7-fae37d6542a0_isrForCrosstalkSources_1125936_23.stderr parsl.dataflow.dflow DEBUG: Task 1430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1430 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1430 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1430 has memoization hash 08e4fa7913de32f75168d05d091c47d6 parsl.dataflow.memoization INFO: Task 1430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b70ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1430 try 0 launched on executor multi with executor id 1431 parsl.dataflow.dflow INFO: Standard output for task 1430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/eb27a66b-8b0f-42c8-9037-434024351594_isrForCrosstalkSources_1125945_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/eb27a66b-8b0f-42c8-9037-434024351594_isrForCrosstalkSources_1125945_43.stderr parsl.dataflow.dflow DEBUG: Task 1431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1431 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1431 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1431 has memoization hash 147d0cd92352b43092522f6e3b67d4a4 parsl.dataflow.memoization INFO: Task 1431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1431 try 0 launched on executor multi with executor id 1432 parsl.dataflow.dflow INFO: Standard output for task 1431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/b66689b4-a84e-40e3-a667-a1e6548304ca_isrForCrosstalkSources_1125925_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/b66689b4-a84e-40e3-a667-a1e6548304ca_isrForCrosstalkSources_1125925_61.stderr parsl.dataflow.dflow DEBUG: Task 1432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1432 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1432 has memoization hash 7e4e425dd6739824e5a21f73c5003f42 parsl.dataflow.memoization INFO: Task 1432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1432 try 0 launched on executor multi with executor id 1433 parsl.dataflow.dflow INFO: Standard output for task 1432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/42da528b-54f7-4f15-890d-741252b67356_isrForCrosstalkSources_1125926_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/42da528b-54f7-4f15-890d-741252b67356_isrForCrosstalkSources_1125926_49.stderr parsl.dataflow.dflow DEBUG: Task 1433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1433 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1433 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1433 has memoization hash 27244f0de6155df8e3460bbecb2f8926 parsl.dataflow.memoization INFO: Task 1433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b723e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1433 try 0 launched on executor multi with executor id 1434 parsl.dataflow.dflow INFO: Standard output for task 1433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e052108a-79f3-45d7-a217-b1a19519bd21_isrForCrosstalkSources_1125901_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e052108a-79f3-45d7-a217-b1a19519bd21_isrForCrosstalkSources_1125901_2.stderr parsl.dataflow.dflow DEBUG: Task 1434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1434 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1434 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1434 has memoization hash ded80ce767420affebb6128a06af83ba parsl.dataflow.memoization INFO: Task 1434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1434 try 0 launched on executor multi with executor id 1435 parsl.dataflow.dflow INFO: Standard output for task 1434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/f7c878c0-e0f5-4766-864b-4475401670a5_isrForCrosstalkSources_1125940_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/f7c878c0-e0f5-4766-864b-4475401670a5_isrForCrosstalkSources_1125940_57.stderr parsl.dataflow.dflow DEBUG: Task 1435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1435 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1435 has memoization hash 0e4a0a2126fe5c9628dd61f99b02a35b parsl.dataflow.memoization INFO: Task 1435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1435 try 0 launched on executor multi with executor id 1436 parsl.dataflow.dflow INFO: Standard output for task 1435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f9e159e2-89c4-4eb4-9021-7b9517c6d635_isrForCrosstalkSources_1125894_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f9e159e2-89c4-4eb4-9021-7b9517c6d635_isrForCrosstalkSources_1125894_42.stderr parsl.dataflow.dflow DEBUG: Task 1436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1436 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1436 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1436 has memoization hash cdffc8bd2f413375258e16cddc292780 parsl.dataflow.memoization INFO: Task 1436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1436 try 0 launched on executor multi with executor id 1437 parsl.dataflow.dflow INFO: Standard output for task 1436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/8149d404-e66a-4489-8a4d-db7d58bbe7ad_isrForCrosstalkSources_1125914_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/8149d404-e66a-4489-8a4d-db7d58bbe7ad_isrForCrosstalkSources_1125914_56.stderr parsl.dataflow.dflow DEBUG: Task 1437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1437 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1437 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1437 has memoization hash 5ba5e0414bcdb0eae80f0560219045b8 parsl.dataflow.memoization INFO: Task 1437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1437 try 0 launched on executor multi with executor id 1438 parsl.dataflow.dflow INFO: Standard output for task 1437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/7ededb62-216b-48fa-8a71-edc07919c426_isrForCrosstalkSources_1125940_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/7ededb62-216b-48fa-8a71-edc07919c426_isrForCrosstalkSources_1125940_24.stderr parsl.dataflow.dflow DEBUG: Task 1438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1438 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1438 has memoization hash a4bb36cd34e46bda793f0f2e955c04a9 parsl.dataflow.memoization INFO: Task 1438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1438 try 0 launched on executor multi with executor id 1439 parsl.dataflow.dflow INFO: Standard output for task 1438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/b94a94d2-88f5-4ec4-892f-2a01bcf355c9_isrForCrosstalkSources_1125898_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/b94a94d2-88f5-4ec4-892f-2a01bcf355c9_isrForCrosstalkSources_1125898_15.stderr parsl.dataflow.dflow DEBUG: Task 1439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1439 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1439 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1439 has memoization hash 69b0ef1e22b47af3f33bcf9aa40163da parsl.dataflow.memoization INFO: Task 1439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1439 try 0 launched on executor multi with executor id 1440 parsl.dataflow.dflow INFO: Standard output for task 1439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/1783cf9d-4682-4e28-9595-738fe6d2bad5_isrForCrosstalkSources_1125926_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/1783cf9d-4682-4e28-9595-738fe6d2bad5_isrForCrosstalkSources_1125926_35.stderr parsl.dataflow.dflow DEBUG: Task 1440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1440 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1440 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1440 has memoization hash 9d63465228997bf65f445ac61a45dde9 parsl.dataflow.memoization INFO: Task 1440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1440 try 0 launched on executor multi with executor id 1441 parsl.dataflow.dflow INFO: Standard output for task 1440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b1776f1e-a9fa-4dcc-bef8-e31d36053d8e_isrForCrosstalkSources_1125938_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b1776f1e-a9fa-4dcc-bef8-e31d36053d8e_isrForCrosstalkSources_1125938_49.stderr parsl.dataflow.dflow DEBUG: Task 1441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1441 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1441 has memoization hash f293e8e21b556d2decf8e620d1e1c9b3 parsl.dataflow.memoization INFO: Task 1441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1441 try 0 launched on executor multi with executor id 1442 parsl.dataflow.dflow INFO: Standard output for task 1441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/416e83cd-4c20-40cf-8766-5dcd82724596_isrForCrosstalkSources_1125899_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/416e83cd-4c20-40cf-8766-5dcd82724596_isrForCrosstalkSources_1125899_41.stderr parsl.dataflow.dflow DEBUG: Task 1442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1442 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1442 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1442 has memoization hash e9aee546e994a18e0efcb5f134db5148 parsl.dataflow.memoization INFO: Task 1442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1442 try 0 launched on executor multi with executor id 1443 parsl.dataflow.dflow INFO: Standard output for task 1442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/ed946c67-9d54-46eb-8d96-5065d88b548c_isrForCrosstalkSources_1125916_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/ed946c67-9d54-46eb-8d96-5065d88b548c_isrForCrosstalkSources_1125916_59.stderr parsl.dataflow.dflow DEBUG: Task 1443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1443 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1443 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1443 has memoization hash 25a38e58baccd0d3b06a2c75d008021e parsl.dataflow.memoization INFO: Task 1443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1443 try 0 launched on executor multi with executor id 1444 parsl.dataflow.dflow INFO: Standard output for task 1443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/a6a20078-eab6-40e3-845f-30be93ed24dd_isrForCrosstalkSources_1125912_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/a6a20078-eab6-40e3-845f-30be93ed24dd_isrForCrosstalkSources_1125912_49.stderr parsl.dataflow.dflow DEBUG: Task 1444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1444 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1444 has memoization hash df94fa37e228a86668f67f7018788690 parsl.dataflow.memoization INFO: Task 1444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:49 proc_lsst.multi:170 [INFO] [multi] job_id 16896411 parsl.dataflow.dflow INFO: Parsl task 1444 try 0 launched on executor multi with executor id 1445 parsl.dataflow.dflow INFO: Standard output for task 1444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d8585084-9214-4526-94ba-fb915ba6a2bc_isrForCrosstalkSources_1125934_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d8585084-9214-4526-94ba-fb915ba6a2bc_isrForCrosstalkSources_1125934_59.stderr parsl.dataflow.dflow DEBUG: Task 1445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1445 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1445 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1445 has memoization hash 540d2b0f13188066e6a99eac19e5d239 parsl.dataflow.memoization INFO: Task 1445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b12cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) proc_lsst.multi INFO: [multi] job_id 16896411 2024-03-07 15:06:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 parsl.dataflow.dflow INFO: Parsl task 1445 try 0 launched on executor multi with executor id 1446 parsl.dataflow.dflow INFO: Standard output for task 1445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/6e72c7e1-8fa3-447b-991f-0f64dc2f2831_isrForCrosstalkSources_1125946_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/6e72c7e1-8fa3-447b-991f-0f64dc2f2831_isrForCrosstalkSources_1125946_43.stderr parsl.dataflow.dflow DEBUG: Task 1446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1446 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1446 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1446 has memoization hash 7b28f2c4931e844e6cab027885d5ab9a parsl.dataflow.memoization INFO: Task 1446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 15:06:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896411 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896411 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16896411 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-07 15:06:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-07 15:06:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 15:06:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1446 try 0 launched on executor multi with executor id 1447 parsl.dataflow.dflow INFO: Standard output for task 1446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/b2e9fb23-1a05-43a2-b4f6-a752537a4ee9_isrForCrosstalkSources_1125928_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/b2e9fb23-1a05-43a2-b4f6-a752537a4ee9_isrForCrosstalkSources_1125928_30.stderr parsl.dataflow.dflow DEBUG: Task 1447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1447 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1447 has memoization hash f276da838b4d6d38ccde9106ef94229f parsl.dataflow.memoization INFO: Task 1447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b72fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1447 try 0 launched on executor multi with executor id 1448 parsl.dataflow.dflow INFO: Standard output for task 1447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/df9b8254-6462-4ab3-9ec6-8086e6f2965d_isrForCrosstalkSources_1125907_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/df9b8254-6462-4ab3-9ec6-8086e6f2965d_isrForCrosstalkSources_1125907_10.stderr parsl.dataflow.dflow DEBUG: Task 1448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1448 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1448 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1448 has memoization hash 90f7782b5d7cec0524161e028345d1dd parsl.dataflow.memoization INFO: Task 1448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39b73060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1448 try 0 launched on executor multi with executor id 1449 parsl.dataflow.dflow INFO: Standard output for task 1448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/ff3a7339-b938-4fc4-af97-7fdd44eecdcd_isrForCrosstalkSources_1125942_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/ff3a7339-b938-4fc4-af97-7fdd44eecdcd_isrForCrosstalkSources_1125942_7.stderr parsl.dataflow.dflow DEBUG: Task 1449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1449 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1449 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1449 has memoization hash d4b195d6535f38283618e6a9a97f99e0 parsl.dataflow.memoization INFO: Task 1449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1449 try 0 launched on executor multi with executor id 1450 parsl.dataflow.dflow INFO: Standard output for task 1449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e3411b2d-730e-49bd-bf17-9b82c7253e0d_isrForCrosstalkSources_1125929_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e3411b2d-730e-49bd-bf17-9b82c7253e0d_isrForCrosstalkSources_1125929_61.stderr parsl.dataflow.dflow DEBUG: Task 1450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1450 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1450 has memoization hash 060312b283627d651074f5bb301b8975 parsl.dataflow.memoization INFO: Task 1450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1450 try 0 launched on executor multi with executor id 1451 parsl.dataflow.dflow INFO: Standard output for task 1450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/d40ece24-1f68-45ec-bb30-2bb300575c97_isrForCrosstalkSources_1125899_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/d40ece24-1f68-45ec-bb30-2bb300575c97_isrForCrosstalkSources_1125899_27.stderr parsl.dataflow.dflow DEBUG: Task 1451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1451 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1451 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1451 has memoization hash 2ada201e6f904e819b7cb3aabd3d37fb parsl.dataflow.memoization INFO: Task 1451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1451 try 0 launched on executor multi with executor id 1452 parsl.dataflow.dflow INFO: Standard output for task 1451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/d2980e08-e0d9-4ce1-aedf-801c77b604f3_isrForCrosstalkSources_1125906_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/d2980e08-e0d9-4ce1-aedf-801c77b604f3_isrForCrosstalkSources_1125906_5.stderr parsl.dataflow.dflow DEBUG: Task 1452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1452 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1452 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1452 has memoization hash 5ef5fc5b9bd4cfec99d58777d523371c parsl.dataflow.memoization INFO: Task 1452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1452 try 0 launched on executor multi with executor id 1453 parsl.dataflow.dflow INFO: Standard output for task 1452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/9148f562-5964-47d1-aa9a-71fbcf68cd86_isrForCrosstalkSources_1125914_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/9148f562-5964-47d1-aa9a-71fbcf68cd86_isrForCrosstalkSources_1125914_22.stderr parsl.dataflow.dflow DEBUG: Task 1453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1453 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1453 has memoization hash 6d6ff5662c324b477a3c7b6e8238196a parsl.dataflow.memoization INFO: Task 1453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1453 try 0 launched on executor multi with executor id 1454 parsl.dataflow.dflow INFO: Standard output for task 1453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/aa01237a-f63a-406a-adc6-7219224648bf_isrForCrosstalkSources_1125907_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/aa01237a-f63a-406a-adc6-7219224648bf_isrForCrosstalkSources_1125907_47.stderr parsl.dataflow.dflow DEBUG: Task 1454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1454 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1454 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1454 has memoization hash b3e4a4b79801c4e92c7bb1fb3be92142 parsl.dataflow.memoization INFO: Task 1454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1454 try 0 launched on executor multi with executor id 1455 parsl.dataflow.dflow INFO: Standard output for task 1454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/95b7b277-2b8e-4dfd-97d1-d74e7b5e7c2f_isrForCrosstalkSources_1125896_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/95b7b277-2b8e-4dfd-97d1-d74e7b5e7c2f_isrForCrosstalkSources_1125896_16.stderr parsl.dataflow.dflow DEBUG: Task 1455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1455 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1455 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1455 has memoization hash 1680d4814ea202a1e02c518ead87aa33 parsl.dataflow.memoization INFO: Task 1455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1455 try 0 launched on executor multi with executor id 1456 parsl.dataflow.dflow INFO: Standard output for task 1455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/ee0f86ed-977c-49b8-a219-03bcd3dc1e47_isrForCrosstalkSources_1125942_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/ee0f86ed-977c-49b8-a219-03bcd3dc1e47_isrForCrosstalkSources_1125942_25.stderr parsl.dataflow.dflow DEBUG: Task 1456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1456 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1456 has memoization hash b4baec6aa547b143fe3646a9361481bb parsl.dataflow.memoization INFO: Task 1456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1456 try 0 launched on executor multi with executor id 1457 parsl.dataflow.dflow INFO: Standard output for task 1456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/3855454e-39b9-4daa-b694-061f338a3ee7_isrForCrosstalkSources_1125932_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/3855454e-39b9-4daa-b694-061f338a3ee7_isrForCrosstalkSources_1125932_6.stderr parsl.dataflow.dflow DEBUG: Task 1457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1457 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1457 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1457 has memoization hash fd2eec2d27357d4b0921139f194687ef parsl.dataflow.memoization INFO: Task 1457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1457 try 0 launched on executor multi with executor id 1458 parsl.dataflow.dflow INFO: Standard output for task 1457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/d5097aeb-f4d2-4dcb-807a-4fb4f09ffdbe_isrForCrosstalkSources_1125898_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/d5097aeb-f4d2-4dcb-807a-4fb4f09ffdbe_isrForCrosstalkSources_1125898_49.stderr parsl.dataflow.dflow DEBUG: Task 1458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1458 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1458 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1458 has memoization hash 778b32a314f3668c3637079e75901e87 parsl.dataflow.memoization INFO: Task 1458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1458 try 0 launched on executor multi with executor id 1459 parsl.dataflow.dflow INFO: Standard output for task 1458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/8a86f2d4-f69d-4f82-9ad1-1c28893f699d_isrForCrosstalkSources_1125942_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/8a86f2d4-f69d-4f82-9ad1-1c28893f699d_isrForCrosstalkSources_1125942_13.stderr parsl.dataflow.dflow DEBUG: Task 1459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1459 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1459 has memoization hash d169f1ce270f071d4febc9e4050d7f55 parsl.dataflow.memoization INFO: Task 1459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1459 try 0 launched on executor multi with executor id 1460 parsl.dataflow.dflow INFO: Standard output for task 1459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/85dd2bd9-ec9a-4268-8b18-d3f4c624f8f0_isrForCrosstalkSources_1125898_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/85dd2bd9-ec9a-4268-8b18-d3f4c624f8f0_isrForCrosstalkSources_1125898_37.stderr parsl.dataflow.dflow DEBUG: Task 1460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1460 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1460 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1460 has memoization hash 3ecae06b5f5143070bff5a30d33b2ce8 parsl.dataflow.memoization INFO: Task 1460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1460 try 0 launched on executor multi with executor id 1461 parsl.dataflow.dflow INFO: Standard output for task 1460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/81c051ed-3aec-49d4-9cf9-f44aec9f5d95_isrForCrosstalkSources_1125934_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/81c051ed-3aec-49d4-9cf9-f44aec9f5d95_isrForCrosstalkSources_1125934_21.stderr parsl.dataflow.dflow DEBUG: Task 1461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1461 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1461 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1461 has memoization hash 29a7562b7cb0c875d0fdf45036294519 parsl.dataflow.memoization INFO: Task 1461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1461 try 0 launched on executor multi with executor id 1462 parsl.dataflow.dflow INFO: Standard output for task 1461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/4943839c-be8f-4f66-9d74-800131a89117_isrForCrosstalkSources_1125943_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/4943839c-be8f-4f66-9d74-800131a89117_isrForCrosstalkSources_1125943_26.stderr parsl.dataflow.dflow DEBUG: Task 1462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1462 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1462 has memoization hash 6522b72eed7793108bd8cc82bdf155c4 parsl.dataflow.memoization INFO: Task 1462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1462 try 0 launched on executor multi with executor id 1463 parsl.dataflow.dflow INFO: Standard output for task 1462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/2749c00c-c1be-426a-9927-7825a2ef7901_isrForCrosstalkSources_1125904_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/2749c00c-c1be-426a-9927-7825a2ef7901_isrForCrosstalkSources_1125904_53.stderr parsl.dataflow.dflow DEBUG: Task 1463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1463 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1463 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1463 has memoization hash 0f241f63bf2c95df920948ab8163dd2a parsl.dataflow.memoization INFO: Task 1463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1463 try 0 launched on executor multi with executor id 1464 parsl.dataflow.dflow INFO: Standard output for task 1463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/c0179c14-e195-4cc8-a450-2f3d5f0445e9_isrForCrosstalkSources_1125925_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/c0179c14-e195-4cc8-a450-2f3d5f0445e9_isrForCrosstalkSources_1125925_40.stderr parsl.dataflow.dflow DEBUG: Task 1464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1464 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1464 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1464 has memoization hash 03f3d46fc8b82fa3d53be7f05f9e5712 parsl.dataflow.memoization INFO: Task 1464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1464 try 0 launched on executor multi with executor id 1465 parsl.dataflow.dflow INFO: Standard output for task 1464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/2febfe70-8bb1-4a94-a312-ba0513a737b1_isrForCrosstalkSources_1125916_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/2febfe70-8bb1-4a94-a312-ba0513a737b1_isrForCrosstalkSources_1125916_30.stderr parsl.dataflow.dflow DEBUG: Task 1465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1465 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1465 has memoization hash b27720ae4e03bf49961590fffb094a35 parsl.dataflow.memoization INFO: Task 1465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1465 try 0 launched on executor multi with executor id 1466 parsl.dataflow.dflow INFO: Standard output for task 1465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/2bb730a9-0798-48f7-be97-6c28d8a5e49a_isrForCrosstalkSources_1125947_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/2bb730a9-0798-48f7-be97-6c28d8a5e49a_isrForCrosstalkSources_1125947_58.stderr parsl.dataflow.dflow DEBUG: Task 1466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1466 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1466 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1466 has memoization hash 6024cf70fdf3ba41f7f5b739828f0273 parsl.dataflow.memoization INFO: Task 1466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1466 try 0 launched on executor multi with executor id 1467 parsl.dataflow.dflow INFO: Standard output for task 1466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/7d3fb1ba-9b4f-4e42-8066-30411a15da31_isrForCrosstalkSources_1125937_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/7d3fb1ba-9b4f-4e42-8066-30411a15da31_isrForCrosstalkSources_1125937_10.stderr parsl.dataflow.dflow DEBUG: Task 1467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1467 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1467 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1467 has memoization hash 7583fab85571534a322127a24c0e2df7 parsl.dataflow.memoization INFO: Task 1467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1467 try 0 launched on executor multi with executor id 1468 parsl.dataflow.dflow INFO: Standard output for task 1467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/8568bf08-4335-46db-9e6a-b27a73277567_isrForCrosstalkSources_1125946_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/8568bf08-4335-46db-9e6a-b27a73277567_isrForCrosstalkSources_1125946_6.stderr parsl.dataflow.dflow DEBUG: Task 1468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1468 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1468 has memoization hash 42ca893a4336bd05be16fdc3e5c11430 parsl.dataflow.memoization INFO: Task 1468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1468 try 0 launched on executor multi with executor id 1469 parsl.dataflow.dflow INFO: Standard output for task 1468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a00f8c53-1e92-4964-bf13-4518db91611b_isrForCrosstalkSources_1125937_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a00f8c53-1e92-4964-bf13-4518db91611b_isrForCrosstalkSources_1125937_18.stderr parsl.dataflow.dflow DEBUG: Task 1469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1469 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1469 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1469 has memoization hash a53e5737b4694f27a122fb5c42af7778 parsl.dataflow.memoization INFO: Task 1469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1469 try 0 launched on executor multi with executor id 1470 parsl.dataflow.dflow INFO: Standard output for task 1469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/c003e0cf-918c-4ff4-9ecf-65956f068c95_isrForCrosstalkSources_1125908_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/c003e0cf-918c-4ff4-9ecf-65956f068c95_isrForCrosstalkSources_1125908_58.stderr parsl.dataflow.dflow DEBUG: Task 1470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1470 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1470 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1470 has memoization hash b15ccb311ebfb1577fb51bb1acaac1c2 parsl.dataflow.memoization INFO: Task 1470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1470 try 0 launched on executor multi with executor id 1471 parsl.dataflow.dflow INFO: Standard output for task 1470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/fe5f9fb0-a1cb-4283-9a48-f1ac00e6180c_isrForCrosstalkSources_1125931_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/fe5f9fb0-a1cb-4283-9a48-f1ac00e6180c_isrForCrosstalkSources_1125931_5.stderr parsl.dataflow.dflow DEBUG: Task 1471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1471 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1471 has memoization hash c76da174b7ba651d7e4963420c7dadb9 parsl.dataflow.memoization INFO: Task 1471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1471 try 0 launched on executor multi with executor id 1472 parsl.dataflow.dflow INFO: Standard output for task 1471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/6a06123d-c633-4f11-8ce6-a407d8c613ca_isrForCrosstalkSources_1125897_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/6a06123d-c633-4f11-8ce6-a407d8c613ca_isrForCrosstalkSources_1125897_14.stderr parsl.dataflow.dflow DEBUG: Task 1472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1472 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1472 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1472 has memoization hash 2d44a3f36065f0cba416f3cf89e9c66a parsl.dataflow.memoization INFO: Task 1472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1472 try 0 launched on executor multi with executor id 1473 parsl.dataflow.dflow INFO: Standard output for task 1472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/3a17492a-6eca-4830-bdd5-4f2b5b8f6ae2_isrForCrosstalkSources_1125934_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/3a17492a-6eca-4830-bdd5-4f2b5b8f6ae2_isrForCrosstalkSources_1125934_56.stderr parsl.dataflow.dflow DEBUG: Task 1473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1473 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1473 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1473 has memoization hash d1bd44340f06dfa94a2e01d525928b14 parsl.dataflow.memoization INFO: Task 1473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1473 try 0 launched on executor multi with executor id 1474 parsl.dataflow.dflow INFO: Standard output for task 1473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d5dc0c21-dc12-4b71-913f-95ecf5cb30b5_isrForCrosstalkSources_1125911_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d5dc0c21-dc12-4b71-913f-95ecf5cb30b5_isrForCrosstalkSources_1125911_58.stderr parsl.dataflow.dflow DEBUG: Task 1474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1474 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1474 has memoization hash 79a9d924fc3a5b9f76f695bec6b74bbd parsl.dataflow.memoization INFO: Task 1474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1474 try 0 launched on executor multi with executor id 1475 parsl.dataflow.dflow INFO: Standard output for task 1474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/09687c14-90a7-41d4-adf0-ef5e99df3dd0_isrForCrosstalkSources_1125935_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/09687c14-90a7-41d4-adf0-ef5e99df3dd0_isrForCrosstalkSources_1125935_18.stderr parsl.dataflow.dflow DEBUG: Task 1475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1475 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1475 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1475 has memoization hash 8f67a76cb6e03bbd8cc6bee223db5d4c parsl.dataflow.memoization INFO: Task 1475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1475 try 0 launched on executor multi with executor id 1476 parsl.dataflow.dflow INFO: Standard output for task 1475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/1d184128-e9da-4a26-94b2-496d95ec1b3e_isrForCrosstalkSources_1125898_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/1d184128-e9da-4a26-94b2-496d95ec1b3e_isrForCrosstalkSources_1125898_3.stderr parsl.dataflow.dflow DEBUG: Task 1476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1476 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1476 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1476 has memoization hash ec5b8a871b113ea2873c35ae103f86e1 parsl.dataflow.memoization INFO: Task 1476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1476 try 0 launched on executor multi with executor id 1477 parsl.dataflow.dflow INFO: Standard output for task 1476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/e72e8fd5-a942-4c0b-bb42-dc63eab26be2_isrForCrosstalkSources_1125890_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/e72e8fd5-a942-4c0b-bb42-dc63eab26be2_isrForCrosstalkSources_1125890_52.stderr parsl.dataflow.dflow DEBUG: Task 1477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1477 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1477 has memoization hash 3462554fc26458d3918faaf4ce300a86 parsl.dataflow.memoization INFO: Task 1477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1477 try 0 launched on executor multi with executor id 1478 parsl.dataflow.dflow INFO: Standard output for task 1477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/d5ac29bb-f7ab-4062-a979-c1c6ede28931_isrForCrosstalkSources_1125910_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/d5ac29bb-f7ab-4062-a979-c1c6ede28931_isrForCrosstalkSources_1125910_49.stderr parsl.dataflow.dflow DEBUG: Task 1478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1478 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1478 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1478 has memoization hash 18167a8aa40bfe827f1734d244cb9a95 parsl.dataflow.memoization INFO: Task 1478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1478 try 0 launched on executor multi with executor id 1479 parsl.dataflow.dflow INFO: Standard output for task 1478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/a9b47bdc-fd48-4a29-9797-6868acc3fb05_isrForCrosstalkSources_1125918_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/a9b47bdc-fd48-4a29-9797-6868acc3fb05_isrForCrosstalkSources_1125918_46.stderr parsl.dataflow.dflow DEBUG: Task 1479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1479 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1479 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1479 has memoization hash fd73744f142186c158a8a648bb685c02 parsl.dataflow.memoization INFO: Task 1479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1479 try 0 launched on executor multi with executor id 1480 parsl.dataflow.dflow INFO: Standard output for task 1479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/fd3b4dcf-48c8-4b14-884d-666fcf181399_isrForCrosstalkSources_1125933_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/fd3b4dcf-48c8-4b14-884d-666fcf181399_isrForCrosstalkSources_1125933_17.stderr parsl.dataflow.dflow DEBUG: Task 1480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1480 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1480 has memoization hash 36e3e2f1e6daeaa5a41850c0e8652c54 parsl.dataflow.memoization INFO: Task 1480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1480 try 0 launched on executor multi with executor id 1481 parsl.dataflow.dflow INFO: Standard output for task 1480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/674a7ce4-d172-4da7-97a6-b663b7825f20_isrForCrosstalkSources_1125924_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/674a7ce4-d172-4da7-97a6-b663b7825f20_isrForCrosstalkSources_1125924_21.stderr parsl.dataflow.dflow DEBUG: Task 1481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1481 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1481 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1481 has memoization hash 3236a14d12b6f277d4630e5b38770225 parsl.dataflow.memoization INFO: Task 1481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1481 try 0 launched on executor multi with executor id 1482 parsl.dataflow.dflow INFO: Standard output for task 1481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/dd3d3710-0776-4b07-bd9e-ea4dde547408_isrForCrosstalkSources_1125916_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/dd3d3710-0776-4b07-bd9e-ea4dde547408_isrForCrosstalkSources_1125916_2.stderr parsl.dataflow.dflow DEBUG: Task 1482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1482 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1482 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1482 has memoization hash fa3e258fb4ed973f721a56fe4be16ca4 parsl.dataflow.memoization INFO: Task 1482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1482 try 0 launched on executor multi with executor id 1483 parsl.dataflow.dflow INFO: Standard output for task 1482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/af3ab0da-f791-4bc1-95ba-9f56c6fbce41_isrForCrosstalkSources_1125904_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/af3ab0da-f791-4bc1-95ba-9f56c6fbce41_isrForCrosstalkSources_1125904_56.stderr parsl.dataflow.dflow DEBUG: Task 1483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1483 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1483 has memoization hash 9d3f46f019017fc76c439622b3567ad9 parsl.dataflow.memoization INFO: Task 1483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1483 try 0 launched on executor multi with executor id 1484 parsl.dataflow.dflow INFO: Standard output for task 1483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/5fcbdd50-d586-4967-b4da-32df4bcde12d_isrForCrosstalkSources_1125892_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/5fcbdd50-d586-4967-b4da-32df4bcde12d_isrForCrosstalkSources_1125892_51.stderr parsl.dataflow.dflow DEBUG: Task 1484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1484 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1484 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1484 has memoization hash 0222deebf61b25e83682d219baa871dc parsl.dataflow.memoization INFO: Task 1484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:50 proc_lsst.multi:170 [INFO] [multi] job_id 16896412 proc_lsst.multi INFO: [multi] job_id 16896412 2024-03-07 15:06:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 15:06:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896412 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896412 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16896412 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-07 15:06:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-07 15:06:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 15:06:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1484 try 0 launched on executor multi with executor id 1485 parsl.dataflow.dflow INFO: Standard output for task 1484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/a98db295-010d-4b99-9894-769905a96f8c_isrForCrosstalkSources_1125944_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/a98db295-010d-4b99-9894-769905a96f8c_isrForCrosstalkSources_1125944_57.stderr parsl.dataflow.dflow DEBUG: Task 1485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1485 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1485 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1485 has memoization hash 63f67723f927b7ade2ca17cca1413956 parsl.dataflow.memoization INFO: Task 1485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1485 try 0 launched on executor multi with executor id 1486 parsl.dataflow.dflow INFO: Standard output for task 1485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/593d914a-ff96-40c4-8e0d-9e549de26648_isrForCrosstalkSources_1125889_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/593d914a-ff96-40c4-8e0d-9e549de26648_isrForCrosstalkSources_1125889_8.stderr parsl.dataflow.dflow DEBUG: Task 1486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1486 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1486 has memoization hash e06707e4bfdce8a33df4cedd91f5860b parsl.dataflow.memoization INFO: Task 1486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1486 try 0 launched on executor multi with executor id 1487 parsl.dataflow.dflow INFO: Standard output for task 1486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/d59693e0-eabe-469e-975e-8c29f125575b_isrForCrosstalkSources_1125944_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/d59693e0-eabe-469e-975e-8c29f125575b_isrForCrosstalkSources_1125944_41.stderr parsl.dataflow.dflow DEBUG: Task 1487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1487 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1487 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1487 has memoization hash 1c77559cd266176e38c423a4880eba79 parsl.dataflow.memoization INFO: Task 1487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1487 try 0 launched on executor multi with executor id 1488 parsl.dataflow.dflow INFO: Standard output for task 1487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/7d654db1-6cb5-49be-bd75-3272f6679e73_isrForCrosstalkSources_1125911_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/7d654db1-6cb5-49be-bd75-3272f6679e73_isrForCrosstalkSources_1125911_33.stderr parsl.dataflow.dflow DEBUG: Task 1488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1488 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1488 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1488 has memoization hash aca5cbe8631efd05197f49f015270827 parsl.dataflow.memoization INFO: Task 1488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1488 try 0 launched on executor multi with executor id 1489 parsl.dataflow.dflow INFO: Standard output for task 1488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/d5374de7-7d19-4ee1-a2e1-2f8ed32d754c_isrForCrosstalkSources_1125945_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/d5374de7-7d19-4ee1-a2e1-2f8ed32d754c_isrForCrosstalkSources_1125945_60.stderr parsl.dataflow.dflow DEBUG: Task 1489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1489 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1489 has memoization hash f4e690323fb41df3c463712fa5c1e4af parsl.dataflow.memoization INFO: Task 1489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3b0edd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1489 try 0 launched on executor multi with executor id 1490 parsl.dataflow.dflow INFO: Standard output for task 1489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/c086f50c-878f-44d6-9c00-1111ae65e5ac_isrForCrosstalkSources_1125913_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/c086f50c-878f-44d6-9c00-1111ae65e5ac_isrForCrosstalkSources_1125913_1.stderr parsl.dataflow.dflow DEBUG: Task 1490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1490 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1490 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1490 has memoization hash 80b9b284d7b16ed40e9be682a88c9c84 parsl.dataflow.memoization INFO: Task 1490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1490 try 0 launched on executor multi with executor id 1491 parsl.dataflow.dflow INFO: Standard output for task 1490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/53ecf6bb-ec88-443a-a126-e2aa9c9a2aa7_isrForCrosstalkSources_1125945_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/53ecf6bb-ec88-443a-a126-e2aa9c9a2aa7_isrForCrosstalkSources_1125945_12.stderr parsl.dataflow.dflow DEBUG: Task 1491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1491 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1491 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1491 has memoization hash 522f0cd0c190ff93113dd182b95fc2e5 parsl.dataflow.memoization INFO: Task 1491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1491 try 0 launched on executor multi with executor id 1492 parsl.dataflow.dflow INFO: Standard output for task 1491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/57164fa4-77cd-42d2-a5bc-0239c58915eb_isrForCrosstalkSources_1125914_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/57164fa4-77cd-42d2-a5bc-0239c58915eb_isrForCrosstalkSources_1125914_2.stderr parsl.dataflow.dflow DEBUG: Task 1492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1492 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1492 has memoization hash d717386388be33db28bc6832bac1dba2 parsl.dataflow.memoization INFO: Task 1492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1492 try 0 launched on executor multi with executor id 1493 parsl.dataflow.dflow INFO: Standard output for task 1492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/4e839d56-16fa-4bd7-bfb1-f662114d8dc5_isrForCrosstalkSources_1125895_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/4e839d56-16fa-4bd7-bfb1-f662114d8dc5_isrForCrosstalkSources_1125895_51.stderr parsl.dataflow.dflow DEBUG: Task 1493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1493 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1493 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1493 has memoization hash 279ffa3ec368087ea3c742b7a4e3df85 parsl.dataflow.memoization INFO: Task 1493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1493 try 0 launched on executor multi with executor id 1494 parsl.dataflow.dflow INFO: Standard output for task 1493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/8d629632-2176-4e35-8b4a-7bc72940fac8_isrForCrosstalkSources_1125946_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/8d629632-2176-4e35-8b4a-7bc72940fac8_isrForCrosstalkSources_1125946_30.stderr parsl.dataflow.dflow DEBUG: Task 1494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1494 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1494 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1494 has memoization hash 9724f90fa58bafd7d47b8a75861bb216 parsl.dataflow.memoization INFO: Task 1494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1494 try 0 launched on executor multi with executor id 1495 parsl.dataflow.dflow INFO: Standard output for task 1494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/cfa5dd94-592a-4c39-b1bd-58fb1ca82a97_isrForCrosstalkSources_1125935_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/cfa5dd94-592a-4c39-b1bd-58fb1ca82a97_isrForCrosstalkSources_1125935_26.stderr parsl.dataflow.dflow DEBUG: Task 1495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1495 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1495 has memoization hash fd993a31784e12722cc2bd23b6564e65 parsl.dataflow.memoization INFO: Task 1495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1495 try 0 launched on executor multi with executor id 1496 parsl.dataflow.dflow INFO: Standard output for task 1495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/9477656f-8822-456d-9c20-d1412166a10a_isrForCrosstalkSources_1125910_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/9477656f-8822-456d-9c20-d1412166a10a_isrForCrosstalkSources_1125910_30.stderr parsl.dataflow.dflow DEBUG: Task 1496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1496 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1496 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1496 has memoization hash a1255989858ce87b7214f54c68ef5ea7 parsl.dataflow.memoization INFO: Task 1496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1496 try 0 launched on executor multi with executor id 1497 parsl.dataflow.dflow INFO: Standard output for task 1496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/41ed0c1d-7c44-456c-9165-a42a8262c128_isrForCrosstalkSources_1125896_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/41ed0c1d-7c44-456c-9165-a42a8262c128_isrForCrosstalkSources_1125896_62.stderr parsl.dataflow.dflow DEBUG: Task 1497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1497 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1497 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1497 has memoization hash 47830ecc0023b9cbe42b8472d12dba58 parsl.dataflow.memoization INFO: Task 1497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1497 try 0 launched on executor multi with executor id 1498 parsl.dataflow.dflow INFO: Standard output for task 1497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/cd01063d-8538-49a6-948f-4a6d9ebaff38_isrForCrosstalkSources_1125919_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/cd01063d-8538-49a6-948f-4a6d9ebaff38_isrForCrosstalkSources_1125919_10.stderr parsl.dataflow.dflow DEBUG: Task 1498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1498 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1498 has memoization hash 56850711d724bac817a5af2faf4edc05 parsl.dataflow.memoization INFO: Task 1498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1498 try 0 launched on executor multi with executor id 1499 parsl.dataflow.dflow INFO: Standard output for task 1498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/bcf922f9-5fca-4b93-b7df-18dc44c20fd7_isrForCrosstalkSources_1125942_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/bcf922f9-5fca-4b93-b7df-18dc44c20fd7_isrForCrosstalkSources_1125942_20.stderr parsl.dataflow.dflow DEBUG: Task 1499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1499 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1499 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1499 has memoization hash ec6d38445986d3b7cf1be3ae37f23982 parsl.dataflow.memoization INFO: Task 1499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1499 try 0 launched on executor multi with executor id 1500 parsl.dataflow.dflow INFO: Standard output for task 1499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/095f79e7-ff75-4b3e-b934-8c6e6aba92ae_isrForCrosstalkSources_1125918_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/095f79e7-ff75-4b3e-b934-8c6e6aba92ae_isrForCrosstalkSources_1125918_51.stderr parsl.dataflow.dflow DEBUG: Task 1500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1500 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1500 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1500 has memoization hash 7f9a0f3c5e0825ed63d50b518851a55f parsl.dataflow.memoization INFO: Task 1500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1500 try 0 launched on executor multi with executor id 1501 parsl.dataflow.dflow INFO: Standard output for task 1500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f66faf50-5c6c-4762-9ec9-551178d17155_isrForCrosstalkSources_1125922_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f66faf50-5c6c-4762-9ec9-551178d17155_isrForCrosstalkSources_1125922_11.stderr parsl.dataflow.dflow DEBUG: Task 1501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1501 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1501 has memoization hash a4d839ff9ec51f641fedf19e26f94742 parsl.dataflow.memoization INFO: Task 1501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1501 try 0 launched on executor multi with executor id 1502 parsl.dataflow.dflow INFO: Standard output for task 1501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/3e8f0078-c745-43ee-bdb7-5b2a7c36ad22_isrForCrosstalkSources_1125904_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/3e8f0078-c745-43ee-bdb7-5b2a7c36ad22_isrForCrosstalkSources_1125904_36.stderr parsl.dataflow.dflow DEBUG: Task 1502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1502 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1502 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1502 has memoization hash dd3aa03bebd771cea423e91c9ed5b92c parsl.dataflow.memoization INFO: Task 1502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1502 try 0 launched on executor multi with executor id 1503 parsl.dataflow.dflow INFO: Standard output for task 1502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/2518428f-06d0-4851-9a5c-e00826e7aa13_isrForCrosstalkSources_1125906_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/2518428f-06d0-4851-9a5c-e00826e7aa13_isrForCrosstalkSources_1125906_51.stderr parsl.dataflow.dflow DEBUG: Task 1503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1503 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1503 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1503 has memoization hash b6b7977096f1174c677e82b15fd76310 parsl.dataflow.memoization INFO: Task 1503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1503 try 0 launched on executor multi with executor id 1504 parsl.dataflow.dflow INFO: Standard output for task 1503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/dc71e1ee-011f-4eb2-87d2-27f1f7d596dc_isrForCrosstalkSources_1125915_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/dc71e1ee-011f-4eb2-87d2-27f1f7d596dc_isrForCrosstalkSources_1125915_1.stderr parsl.dataflow.dflow DEBUG: Task 1504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1504 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1504 has memoization hash 5c522e30515d144f7ca00572423ef171 parsl.dataflow.memoization INFO: Task 1504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1504 try 0 launched on executor multi with executor id 1505 parsl.dataflow.dflow INFO: Standard output for task 1504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/27b4e139-6db0-4082-a005-ced4e2078892_isrForCrosstalkSources_1125947_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/27b4e139-6db0-4082-a005-ced4e2078892_isrForCrosstalkSources_1125947_57.stderr parsl.dataflow.dflow DEBUG: Task 1505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1505 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1505 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1505 has memoization hash 259488bc564e71cb45b822f613507e24 parsl.dataflow.memoization INFO: Task 1505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1505 try 0 launched on executor multi with executor id 1506 parsl.dataflow.dflow INFO: Standard output for task 1505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/4edea987-a558-4704-96ba-60eecbd0747a_isrForCrosstalkSources_1125928_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/4edea987-a558-4704-96ba-60eecbd0747a_isrForCrosstalkSources_1125928_15.stderr parsl.dataflow.dflow DEBUG: Task 1506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1506 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1506 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1506 has memoization hash 329f237aa97ea3ae43772c588d39bfb3 parsl.dataflow.memoization INFO: Task 1506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1506 try 0 launched on executor multi with executor id 1507 parsl.dataflow.dflow INFO: Standard output for task 1506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/8d54148b-0c6d-42d2-85bd-81ef459b55fe_isrForCrosstalkSources_1125904_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/8d54148b-0c6d-42d2-85bd-81ef459b55fe_isrForCrosstalkSources_1125904_8.stderr parsl.dataflow.dflow DEBUG: Task 1507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1507 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1507 has memoization hash f5e7742071d74edbd02ff96a1650d4c5 parsl.dataflow.memoization INFO: Task 1507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1507 try 0 launched on executor multi with executor id 1508 parsl.dataflow.dflow INFO: Standard output for task 1507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d03eb2b3-d4be-4d6e-a7db-3bcc5ab9c10e_isrForCrosstalkSources_1125905_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d03eb2b3-d4be-4d6e-a7db-3bcc5ab9c10e_isrForCrosstalkSources_1125905_38.stderr parsl.dataflow.dflow DEBUG: Task 1508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1508 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1508 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1508 has memoization hash 052f2602ccc05c25cfe75e9a8d7cff8f parsl.dataflow.memoization INFO: Task 1508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1508 try 0 launched on executor multi with executor id 1509 parsl.dataflow.dflow INFO: Standard output for task 1508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/dcab9d89-f0dc-4f9d-aaf8-0f2f7663cf93_isrForCrosstalkSources_1125920_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/dcab9d89-f0dc-4f9d-aaf8-0f2f7663cf93_isrForCrosstalkSources_1125920_24.stderr parsl.dataflow.dflow DEBUG: Task 1509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1509 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1509 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1509 has memoization hash 242f5549148cf82ea10e2d56c590e8fc parsl.dataflow.memoization INFO: Task 1509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1509 try 0 launched on executor multi with executor id 1510 parsl.dataflow.dflow INFO: Standard output for task 1509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/4d39bac8-0981-4337-8102-5a903a2692ea_isrForCrosstalkSources_1125904_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/4d39bac8-0981-4337-8102-5a903a2692ea_isrForCrosstalkSources_1125904_14.stderr parsl.dataflow.dflow DEBUG: Task 1510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1510 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1510 has memoization hash 851bfffab539467ed023f294ac6b89fb parsl.dataflow.memoization INFO: Task 1510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1510 try 0 launched on executor multi with executor id 1511 parsl.dataflow.dflow INFO: Standard output for task 1510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/3cf337d3-7a3d-47f6-a718-7de7081dccf3_isrForCrosstalkSources_1125945_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/3cf337d3-7a3d-47f6-a718-7de7081dccf3_isrForCrosstalkSources_1125945_17.stderr parsl.dataflow.dflow DEBUG: Task 1511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1511 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1511 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1511 has memoization hash 862389673e37a233c58d242ff7b4e671 parsl.dataflow.memoization INFO: Task 1511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1511 try 0 launched on executor multi with executor id 1512 parsl.dataflow.dflow INFO: Standard output for task 1511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/bad8542f-8f77-490d-b385-ca29851ea41f_isrForCrosstalkSources_1125933_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/bad8542f-8f77-490d-b385-ca29851ea41f_isrForCrosstalkSources_1125933_37.stderr parsl.dataflow.dflow DEBUG: Task 1512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1512 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1512 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1512 has memoization hash 75bd69b36e20645c2574da72af190ac2 parsl.dataflow.memoization INFO: Task 1512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1512 try 0 launched on executor multi with executor id 1513 parsl.dataflow.dflow INFO: Standard output for task 1512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/4a047409-d56a-4803-b615-63745db0ac8c_isrForCrosstalkSources_1125896_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/4a047409-d56a-4803-b615-63745db0ac8c_isrForCrosstalkSources_1125896_45.stderr parsl.dataflow.dflow DEBUG: Task 1513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1513 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1513 has memoization hash 6d0417b816cb14194bf7e0fd644caeee parsl.dataflow.memoization INFO: Task 1513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1513 try 0 launched on executor multi with executor id 1514 parsl.dataflow.dflow INFO: Standard output for task 1513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/d647a383-2ab0-4da0-b379-10b4260cff82_isrForCrosstalkSources_1125918_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/d647a383-2ab0-4da0-b379-10b4260cff82_isrForCrosstalkSources_1125918_6.stderr parsl.dataflow.dflow DEBUG: Task 1514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1514 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1514 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1514 has memoization hash 4e48f108a93413bf7b7a286ec3a31630 parsl.dataflow.memoization INFO: Task 1514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:50 proc_lsst.multi:170 [INFO] [multi] job_id 16896413 proc_lsst.multi INFO: [multi] job_id 16896413 2024-03-07 15:06:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 15:06:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896413 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896413 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16896413 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-07 15:06:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-07 15:06:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 15:06:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1514 try 0 launched on executor multi with executor id 1515 parsl.dataflow.dflow INFO: Standard output for task 1514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/96bedc43-8a05-405d-b4c9-ca997a5da9ac_isrForCrosstalkSources_1125896_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/96bedc43-8a05-405d-b4c9-ca997a5da9ac_isrForCrosstalkSources_1125896_3.stderr parsl.dataflow.dflow DEBUG: Task 1515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1515 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1515 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1515 has memoization hash ebfaf1801ec02376335d8fe545879bff parsl.dataflow.memoization INFO: Task 1515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1515 try 0 launched on executor multi with executor id 1516 parsl.dataflow.dflow INFO: Standard output for task 1515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d96cf3c4-8536-4e83-84cf-8cac15a6986f_isrForCrosstalkSources_1125919_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d96cf3c4-8536-4e83-84cf-8cac15a6986f_isrForCrosstalkSources_1125919_44.stderr parsl.dataflow.dflow DEBUG: Task 1516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1516 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1516 has memoization hash 08714656d99f64fee3b858bb537fd214 parsl.dataflow.memoization INFO: Task 1516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1516 try 0 launched on executor multi with executor id 1517 parsl.dataflow.dflow INFO: Standard output for task 1516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/30e06081-d6fe-4e31-9424-95d60fdc3b2a_isrForCrosstalkSources_1125897_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/30e06081-d6fe-4e31-9424-95d60fdc3b2a_isrForCrosstalkSources_1125897_42.stderr parsl.dataflow.dflow DEBUG: Task 1517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1517 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1517 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1517 has memoization hash 5e01d572b5ae3bfea063c41bb4181af2 parsl.dataflow.memoization INFO: Task 1517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1517 try 0 launched on executor multi with executor id 1518 parsl.dataflow.dflow INFO: Standard output for task 1517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/d4d44e2f-728f-4f23-8d1d-91c2c4a79372_isrForCrosstalkSources_1125916_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/d4d44e2f-728f-4f23-8d1d-91c2c4a79372_isrForCrosstalkSources_1125916_55.stderr parsl.dataflow.dflow DEBUG: Task 1518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1518 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1518 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1518 has memoization hash d7eba77399b02c702ec7e344b72cbc09 parsl.dataflow.memoization INFO: Task 1518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1518 try 0 launched on executor multi with executor id 1519 parsl.dataflow.dflow INFO: Standard output for task 1518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/fc921be8-519c-440d-be2f-22ec539c2c17_isrForCrosstalkSources_1125942_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/fc921be8-519c-440d-be2f-22ec539c2c17_isrForCrosstalkSources_1125942_4.stderr parsl.dataflow.dflow DEBUG: Task 1519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1519 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1519 has memoization hash 8025bd30395e3d5a0d6fb637bb99b889 parsl.dataflow.memoization INFO: Task 1519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1519 try 0 launched on executor multi with executor id 1520 parsl.dataflow.dflow INFO: Standard output for task 1519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/9c7fd067-9c6f-4bb3-be36-35b875a9cf04_isrForCrosstalkSources_1125891_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/9c7fd067-9c6f-4bb3-be36-35b875a9cf04_isrForCrosstalkSources_1125891_58.stderr parsl.dataflow.dflow DEBUG: Task 1520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1520 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1520 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1520 has memoization hash 3d769216d2c89cbb4114ae687c7dbf61 parsl.dataflow.memoization INFO: Task 1520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1520 try 0 launched on executor multi with executor id 1521 parsl.dataflow.dflow INFO: Standard output for task 1520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/46b75d21-0a64-4f90-b197-984589f8f037_isrForCrosstalkSources_1125906_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/46b75d21-0a64-4f90-b197-984589f8f037_isrForCrosstalkSources_1125906_20.stderr parsl.dataflow.dflow DEBUG: Task 1521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1521 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1521 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1521 has memoization hash 4ebdf884bac3b04558da2592c2303b81 parsl.dataflow.memoization INFO: Task 1521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1521 try 0 launched on executor multi with executor id 1522 parsl.dataflow.dflow INFO: Standard output for task 1521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/2c88b2cb-1263-4853-96a3-848995052a27_isrForCrosstalkSources_1125940_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/2c88b2cb-1263-4853-96a3-848995052a27_isrForCrosstalkSources_1125940_3.stderr parsl.dataflow.dflow DEBUG: Task 1522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1522 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1522 has memoization hash f2724420fd4c6cd14370574729bebe61 parsl.dataflow.memoization INFO: Task 1522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1522 try 0 launched on executor multi with executor id 1523 parsl.dataflow.dflow INFO: Standard output for task 1522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6ce586cf-4e95-4fd5-a438-3996b57a4e07_isrForCrosstalkSources_1125909_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6ce586cf-4e95-4fd5-a438-3996b57a4e07_isrForCrosstalkSources_1125909_1.stderr parsl.dataflow.dflow DEBUG: Task 1523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1523 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1523 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1523 has memoization hash 2ce157c85fea00417f51909edb58374c parsl.dataflow.memoization INFO: Task 1523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1523 try 0 launched on executor multi with executor id 1524 parsl.dataflow.dflow INFO: Standard output for task 1523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/6dfa0ddf-312c-43a4-934e-67ffc2c24f91_isrForCrosstalkSources_1125899_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/6dfa0ddf-312c-43a4-934e-67ffc2c24f91_isrForCrosstalkSources_1125899_33.stderr parsl.dataflow.dflow DEBUG: Task 1524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1524 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1524 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1524 has memoization hash 02220cc91c62daf0907270ff27fa4fe0 parsl.dataflow.memoization INFO: Task 1524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1524 try 0 launched on executor multi with executor id 1525 parsl.dataflow.dflow INFO: Standard output for task 1524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/9dfb14ae-6d64-4d41-9d60-8940180602a7_isrForCrosstalkSources_1125916_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/9dfb14ae-6d64-4d41-9d60-8940180602a7_isrForCrosstalkSources_1125916_46.stderr parsl.dataflow.dflow DEBUG: Task 1525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1525 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1525 has memoization hash cb03d98cf2d556311272f3486511a5a0 parsl.dataflow.memoization INFO: Task 1525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1525 try 0 launched on executor multi with executor id 1526 parsl.dataflow.dflow INFO: Standard output for task 1525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/697fb45f-bcef-483f-9bdf-1d2cad352ac4_isrForCrosstalkSources_1125925_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/697fb45f-bcef-483f-9bdf-1d2cad352ac4_isrForCrosstalkSources_1125925_33.stderr parsl.dataflow.dflow DEBUG: Task 1526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1526 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1526 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1526 has memoization hash 733d21902956c39ad07c9186bccdb65e parsl.dataflow.memoization INFO: Task 1526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1526 try 0 launched on executor multi with executor id 1527 parsl.dataflow.dflow INFO: Standard output for task 1526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/68d38b6a-f777-4e42-9ab7-c48665c39ab1_isrForCrosstalkSources_1125930_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/68d38b6a-f777-4e42-9ab7-c48665c39ab1_isrForCrosstalkSources_1125930_51.stderr parsl.dataflow.dflow DEBUG: Task 1527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1527 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1527 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1527 has memoization hash d5f263783b0cf4fe88395d34d299340f parsl.dataflow.memoization INFO: Task 1527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1527 try 0 launched on executor multi with executor id 1528 parsl.dataflow.dflow INFO: Standard output for task 1527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/fc76317f-ebb9-4c33-9f20-ac7e35e54b05_isrForCrosstalkSources_1125901_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/fc76317f-ebb9-4c33-9f20-ac7e35e54b05_isrForCrosstalkSources_1125901_20.stderr parsl.dataflow.dflow DEBUG: Task 1528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1528 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1528 has memoization hash dabe79eb756d25fbadecb4c4e40ff136 parsl.dataflow.memoization INFO: Task 1528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1528 try 0 launched on executor multi with executor id 1529 parsl.dataflow.dflow INFO: Standard output for task 1528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7ebeba42-00e8-434a-acc0-17eceb52526b_isrForCrosstalkSources_1125929_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7ebeba42-00e8-434a-acc0-17eceb52526b_isrForCrosstalkSources_1125929_25.stderr parsl.dataflow.dflow DEBUG: Task 1529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1529 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1529 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1529 has memoization hash a26b7839087c5d2368e00ed780853bdf parsl.dataflow.memoization INFO: Task 1529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1529 try 0 launched on executor multi with executor id 1530 parsl.dataflow.dflow INFO: Standard output for task 1529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/bcde2489-10eb-4a91-a8e9-cdd72c9c7824_isrForCrosstalkSources_1125911_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/bcde2489-10eb-4a91-a8e9-cdd72c9c7824_isrForCrosstalkSources_1125911_2.stderr parsl.dataflow.dflow DEBUG: Task 1530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1530 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1530 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1530 has memoization hash 8c6a11e3e70a6728d9670ae61dbb571b parsl.dataflow.memoization INFO: Task 1530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1530 try 0 launched on executor multi with executor id 1531 parsl.dataflow.dflow INFO: Standard output for task 1530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/cbe1d291-50fa-47af-8738-05fc06a96f37_isrForCrosstalkSources_1125919_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/cbe1d291-50fa-47af-8738-05fc06a96f37_isrForCrosstalkSources_1125919_34.stderr parsl.dataflow.dflow DEBUG: Task 1531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1531 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1531 has memoization hash a4874b0f552f13ef4773dbe8c85aab74 parsl.dataflow.memoization INFO: Task 1531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1531 try 0 launched on executor multi with executor id 1532 parsl.dataflow.dflow INFO: Standard output for task 1531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/c1aed873-6776-4ed3-bfd6-e8dbeacb7619_isrForCrosstalkSources_1125890_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/c1aed873-6776-4ed3-bfd6-e8dbeacb7619_isrForCrosstalkSources_1125890_42.stderr parsl.dataflow.dflow DEBUG: Task 1532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1532 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1532 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1532 has memoization hash f509a5ab10e82bced4df3f0075980161 parsl.dataflow.memoization INFO: Task 1532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1532 try 0 launched on executor multi with executor id 1533 parsl.dataflow.dflow INFO: Standard output for task 1532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/bb78230d-2215-40a8-9d0d-ee11ef1e6778_isrForCrosstalkSources_1125892_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/bb78230d-2215-40a8-9d0d-ee11ef1e6778_isrForCrosstalkSources_1125892_8.stderr parsl.dataflow.dflow DEBUG: Task 1533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1533 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1533 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1533 has memoization hash 3f20e0b093c244d28aecb97c86577b7a parsl.dataflow.memoization INFO: Task 1533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1533 try 0 launched on executor multi with executor id 1534 parsl.dataflow.dflow INFO: Standard output for task 1533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/cd703c02-faa7-4c93-a389-09e49a06881f_isrForCrosstalkSources_1125944_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/cd703c02-faa7-4c93-a389-09e49a06881f_isrForCrosstalkSources_1125944_1.stderr parsl.dataflow.dflow DEBUG: Task 1534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1534 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1534 has memoization hash 3ebff62b786acc1610e5cbe2d8cec3b1 parsl.dataflow.memoization INFO: Task 1534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1534 try 0 launched on executor multi with executor id 1535 parsl.dataflow.dflow INFO: Standard output for task 1534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/76aaf5a3-195b-46e6-83ba-56bdd165959f_isrForCrosstalkSources_1125889_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/76aaf5a3-195b-46e6-83ba-56bdd165959f_isrForCrosstalkSources_1125889_60.stderr parsl.dataflow.dflow DEBUG: Task 1535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1535 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1535 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1535 has memoization hash 5799dc28d06549a6641bad28e99a0edc parsl.dataflow.memoization INFO: Task 1535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1535 try 0 launched on executor multi with executor id 1536 parsl.dataflow.dflow INFO: Standard output for task 1535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/6f4412db-3730-4730-a676-838759d1b34f_isrForCrosstalkSources_1125925_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/6f4412db-3730-4730-a676-838759d1b34f_isrForCrosstalkSources_1125925_10.stderr parsl.dataflow.dflow DEBUG: Task 1536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1536 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1536 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1536 has memoization hash 61e4cdd6090da921aeca894db3b75ef9 parsl.dataflow.memoization INFO: Task 1536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a340e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1536 try 0 launched on executor multi with executor id 1537 parsl.dataflow.dflow INFO: Standard output for task 1536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/0d4b75ea-f584-49ea-8039-d988bb0a1da0_isrForCrosstalkSources_1125911_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/0d4b75ea-f584-49ea-8039-d988bb0a1da0_isrForCrosstalkSources_1125911_12.stderr parsl.dataflow.dflow DEBUG: Task 1537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1537 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1537 has memoization hash ce9e494f9af8046a6b9c7964541d6b5e parsl.dataflow.memoization INFO: Task 1537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1537 try 0 launched on executor multi with executor id 1538 parsl.dataflow.dflow INFO: Standard output for task 1537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/6a313fab-05d2-4d76-a55a-0d2f75838129_isrForCrosstalkSources_1125946_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/6a313fab-05d2-4d76-a55a-0d2f75838129_isrForCrosstalkSources_1125946_12.stderr parsl.dataflow.dflow DEBUG: Task 1538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1538 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1538 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1538 has memoization hash 0a789998351a6e74dc83cf208931050b parsl.dataflow.memoization INFO: Task 1538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1538 try 0 launched on executor multi with executor id 1539 parsl.dataflow.dflow INFO: Standard output for task 1538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/ef81d17e-c45c-4777-a6b0-d82118b39102_isrForCrosstalkSources_1125898_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/ef81d17e-c45c-4777-a6b0-d82118b39102_isrForCrosstalkSources_1125898_35.stderr parsl.dataflow.dflow DEBUG: Task 1539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1539 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1539 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1539 has memoization hash d90ca67b64cc37fe509a6e890e858bda parsl.dataflow.memoization INFO: Task 1539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a342c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1539 try 0 launched on executor multi with executor id 1540 parsl.dataflow.dflow INFO: Standard output for task 1539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/5ab82810-0fca-4276-860e-6ab43db0b933_isrForCrosstalkSources_1125889_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/5ab82810-0fca-4276-860e-6ab43db0b933_isrForCrosstalkSources_1125889_57.stderr parsl.dataflow.dflow DEBUG: Task 1540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1540 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1540 has memoization hash 6bfc5a734c4e0cbacaddcab294359609 parsl.dataflow.memoization INFO: Task 1540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1540 try 0 launched on executor multi with executor id 1541 parsl.dataflow.dflow INFO: Standard output for task 1540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/3c740822-e5ab-49ca-a892-101b1f1b7e99_isrForCrosstalkSources_1125926_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/3c740822-e5ab-49ca-a892-101b1f1b7e99_isrForCrosstalkSources_1125926_26.stderr parsl.dataflow.dflow DEBUG: Task 1541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1541 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1541 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1541 has memoization hash 81e21c875471d8ba4e544a91f1884949 parsl.dataflow.memoization INFO: Task 1541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1541 try 0 launched on executor multi with executor id 1542 parsl.dataflow.dflow INFO: Standard output for task 1541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/bc2b3823-7bc5-428b-a5ea-4870de867876_isrForCrosstalkSources_1125946_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/bc2b3823-7bc5-428b-a5ea-4870de867876_isrForCrosstalkSources_1125946_15.stderr parsl.dataflow.dflow DEBUG: Task 1542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1542 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1542 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1542 has memoization hash 02276a11a3e7cc55070a14c348af40b9 parsl.dataflow.memoization INFO: Task 1542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1542 try 0 launched on executor multi with executor id 1543 parsl.dataflow.dflow INFO: Standard output for task 1542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/1d2720db-c471-4fff-a5fb-ff25b72accff_isrForCrosstalkSources_1125888_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/1d2720db-c471-4fff-a5fb-ff25b72accff_isrForCrosstalkSources_1125888_27.stderr parsl.dataflow.dflow DEBUG: Task 1543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1543 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1543 has memoization hash d9d787b4a1c699c11834e606782101df parsl.dataflow.memoization INFO: Task 1543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a344a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1543 try 0 launched on executor multi with executor id 1544 parsl.dataflow.dflow INFO: Standard output for task 1543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/23faffcd-6270-4b43-bc2f-6bd258eff2de_isrForCrosstalkSources_1125936_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/23faffcd-6270-4b43-bc2f-6bd258eff2de_isrForCrosstalkSources_1125936_28.stderr parsl.dataflow.dflow DEBUG: Task 1544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1544 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1544 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1544 has memoization hash 574351a7d25cbaafe4b646278583088d parsl.dataflow.memoization INFO: Task 1544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a345e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1544 try 0 launched on executor multi with executor id 1545 parsl.dataflow.dflow INFO: Standard output for task 1544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/43db868b-327b-4390-9f67-af0660193070_isrForCrosstalkSources_1125919_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/43db868b-327b-4390-9f67-af0660193070_isrForCrosstalkSources_1125919_52.stderr parsl.dataflow.dflow DEBUG: Task 1545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1545 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1545 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1545 has memoization hash 1699d4f1c31feb62b5f40648fdc881d7 parsl.dataflow.memoization INFO: Task 1545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1545 try 0 launched on executor multi with executor id 1546 parsl.dataflow.dflow INFO: Standard output for task 1545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/f2df39d4-d78f-4218-889e-1ab121e07c03_isrForCrosstalkSources_1125912_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/f2df39d4-d78f-4218-889e-1ab121e07c03_isrForCrosstalkSources_1125912_11.stderr parsl.dataflow.dflow DEBUG: Task 1546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1546 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1546 has memoization hash b78d9563795cf01c66524e51900a16d0 parsl.dataflow.memoization INFO: Task 1546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1546 try 0 launched on executor multi with executor id 1547 parsl.dataflow.dflow INFO: Standard output for task 1546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/e4d6fd2e-b8f8-4a46-b8a1-96f2bc77ba99_isrForCrosstalkSources_1125947_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/e4d6fd2e-b8f8-4a46-b8a1-96f2bc77ba99_isrForCrosstalkSources_1125947_49.stderr parsl.dataflow.dflow DEBUG: Task 1547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1547 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1547 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1547 has memoization hash 05854fb761c44ddbacb183c5ad805fbf parsl.dataflow.memoization INFO: Task 1547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1547 try 0 launched on executor multi with executor id 1548 parsl.dataflow.dflow INFO: Standard output for task 1547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/431202fc-ea44-414e-a1a6-23c93b8d627d_isrForCrosstalkSources_1125921_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/431202fc-ea44-414e-a1a6-23c93b8d627d_isrForCrosstalkSources_1125921_30.stderr parsl.dataflow.dflow DEBUG: Task 1548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1548 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1548 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1548 has memoization hash c02cfddf41a2bde784ff44a924d3f6e2 parsl.dataflow.memoization INFO: Task 1548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a347c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1548 try 0 launched on executor multi with executor id 1549 parsl.dataflow.dflow INFO: Standard output for task 1548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/165f3470-29c6-44e3-8e33-9c53969765eb_isrForCrosstalkSources_1125926_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/165f3470-29c6-44e3-8e33-9c53969765eb_isrForCrosstalkSources_1125926_16.stderr parsl.dataflow.dflow DEBUG: Task 1549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1549 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1549 has memoization hash aab00952b4bf71f768b875667e58f3df parsl.dataflow.memoization INFO: Task 1549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1549 try 0 launched on executor multi with executor id 1550 parsl.dataflow.dflow INFO: Standard output for task 1549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/55acf85f-9cfb-4c59-bcec-bd261ea08a2b_isrForCrosstalkSources_1125913_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/55acf85f-9cfb-4c59-bcec-bd261ea08a2b_isrForCrosstalkSources_1125913_5.stderr parsl.dataflow.dflow DEBUG: Task 1550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1550 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1550 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1550 has memoization hash d7d9086513a6f4a055ba4913f49d4e9c parsl.dataflow.memoization INFO: Task 1550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a349a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1550 try 0 launched on executor multi with executor id 1551 parsl.dataflow.dflow INFO: Standard output for task 1550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c89347a9-f3d5-4a54-915d-4a4aba90ba37_isrForCrosstalkSources_1125917_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c89347a9-f3d5-4a54-915d-4a4aba90ba37_isrForCrosstalkSources_1125917_34.stderr parsl.dataflow.dflow DEBUG: Task 1551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1551 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1551 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1551 has memoization hash 908449b04a4cfdb1075063bcc4fc9019 parsl.dataflow.memoization INFO: Task 1551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1551 try 0 launched on executor multi with executor id 1552 parsl.dataflow.dflow INFO: Standard output for task 1551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/68c8b0ae-95aa-4af3-894c-d5deb741f55d_isrForCrosstalkSources_1125922_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/68c8b0ae-95aa-4af3-894c-d5deb741f55d_isrForCrosstalkSources_1125922_30.stderr parsl.dataflow.dflow DEBUG: Task 1552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1552 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1552 has memoization hash 506816c8c9a1168e3829aad624a907d6 parsl.dataflow.memoization INFO: Task 1552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1552 try 0 launched on executor multi with executor id 1553 parsl.dataflow.dflow INFO: Standard output for task 1552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/e099f92d-722b-4803-a9fa-74fba2d14cc0_isrForCrosstalkSources_1125944_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/e099f92d-722b-4803-a9fa-74fba2d14cc0_isrForCrosstalkSources_1125944_53.stderr parsl.dataflow.dflow DEBUG: Task 1553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1553 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1553 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1553 has memoization hash 33d731ab4b16bba2efb8fc4d7c5eece1 parsl.dataflow.memoization INFO: Task 1553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1553 try 0 launched on executor multi with executor id 1554 parsl.dataflow.dflow INFO: Standard output for task 1553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/bff942ce-6973-44e1-89f6-e1630d6a05da_isrForCrosstalkSources_1125902_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/bff942ce-6973-44e1-89f6-e1630d6a05da_isrForCrosstalkSources_1125902_38.stderr parsl.dataflow.dflow DEBUG: Task 1554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1554 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1554 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1554 has memoization hash 317e71ee379820d88de860c0e3349ae9 parsl.dataflow.memoization INFO: Task 1554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1554 try 0 launched on executor multi with executor id 1555 parsl.dataflow.dflow INFO: Standard output for task 1554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8182d01b-b18a-44cf-a51c-4dd8d5ece67a_isrForCrosstalkSources_1125936_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8182d01b-b18a-44cf-a51c-4dd8d5ece67a_isrForCrosstalkSources_1125936_43.stderr parsl.dataflow.dflow DEBUG: Task 1555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1555 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1555 has memoization hash d488ffe3c92525bc935f9a6e65fc5272 parsl.dataflow.memoization INFO: Task 1555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1555 try 0 launched on executor multi with executor id 1556 parsl.dataflow.dflow INFO: Standard output for task 1555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/04b49bdb-7fb2-4831-b626-e1452dd22b8b_isrForCrosstalkSources_1125912_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/04b49bdb-7fb2-4831-b626-e1452dd22b8b_isrForCrosstalkSources_1125912_23.stderr parsl.dataflow.dflow DEBUG: Task 1556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1556 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1556 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1556 has memoization hash 345ecc2ff667085245cbd46f777fc599 parsl.dataflow.memoization INFO: Task 1556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1556 try 0 launched on executor multi with executor id 1557 parsl.dataflow.dflow INFO: Standard output for task 1556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/3c53ba15-795c-4375-bad6-80d74a365160_isrForCrosstalkSources_1125926_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/3c53ba15-795c-4375-bad6-80d74a365160_isrForCrosstalkSources_1125926_17.stderr parsl.dataflow.dflow DEBUG: Task 1557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1557 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1557 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1557 has memoization hash 8b48ab2e935617724564536117989f5b parsl.dataflow.memoization INFO: Task 1557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1557 try 0 launched on executor multi with executor id 1558 parsl.dataflow.dflow INFO: Standard output for task 1557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/124fc6f9-5259-4d5d-b289-1b7bc539a22b_isrForCrosstalkSources_1125922_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/124fc6f9-5259-4d5d-b289-1b7bc539a22b_isrForCrosstalkSources_1125922_23.stderr parsl.dataflow.dflow DEBUG: Task 1558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1558 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1558 has memoization hash 41e6d0c8ed2086dda4320b64d999f60a parsl.dataflow.memoization INFO: Task 1558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1558 try 0 launched on executor multi with executor id 1559 parsl.dataflow.dflow INFO: Standard output for task 1558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/464e6e95-b412-468d-9723-95068817b8a0_isrForCrosstalkSources_1125911_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/464e6e95-b412-468d-9723-95068817b8a0_isrForCrosstalkSources_1125911_14.stderr parsl.dataflow.dflow DEBUG: Task 1559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1559 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1559 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1559 has memoization hash 7d4a02b7432abbcce8fcc3ef329fed87 parsl.dataflow.memoization INFO: Task 1559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1559 try 0 launched on executor multi with executor id 1560 parsl.dataflow.dflow INFO: Standard output for task 1559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/93075154-b142-4556-ab2d-fb00fb0fe242_isrForCrosstalkSources_1125915_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/93075154-b142-4556-ab2d-fb00fb0fe242_isrForCrosstalkSources_1125915_45.stderr parsl.dataflow.dflow DEBUG: Task 1560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1560 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1560 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1560 has memoization hash 721f1b906ff0cd69e75eeb977dad5176 parsl.dataflow.memoization INFO: Task 1560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1560 try 0 launched on executor multi with executor id 1561 parsl.dataflow.dflow INFO: Standard output for task 1560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/8cf19534-b069-48f1-a7d6-6004df818572_isrForCrosstalkSources_1125927_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/8cf19534-b069-48f1-a7d6-6004df818572_isrForCrosstalkSources_1125927_60.stderr parsl.dataflow.dflow DEBUG: Task 1561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1561 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1561 has memoization hash 4d0c1d3188fa12a7d9ae4c947b436494 parsl.dataflow.memoization INFO: Task 1561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1561 try 0 launched on executor multi with executor id 1562 parsl.dataflow.dflow INFO: Standard output for task 1561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/6d641af9-776e-4403-b740-a985071b0ceb_isrForCrosstalkSources_1125907_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/6d641af9-776e-4403-b740-a985071b0ceb_isrForCrosstalkSources_1125907_53.stderr parsl.dataflow.dflow DEBUG: Task 1562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1562 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1562 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1562 has memoization hash 889ff74a4c71d5816a92d983edef1bd2 parsl.dataflow.memoization INFO: Task 1562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a351c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1562 try 0 launched on executor multi with executor id 1563 parsl.dataflow.dflow INFO: Standard output for task 1562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/48ca9569-897f-4ec4-849c-38f4d155f6ee_isrForCrosstalkSources_1125919_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/48ca9569-897f-4ec4-849c-38f4d155f6ee_isrForCrosstalkSources_1125919_24.stderr parsl.dataflow.dflow DEBUG: Task 1563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1563 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1563 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1563 has memoization hash 73294469d2cb352842a5d1405a0c00e7 parsl.dataflow.memoization INFO: Task 1563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1563 try 0 launched on executor multi with executor id 1564 parsl.dataflow.dflow INFO: Standard output for task 1563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/7dd49a0a-1238-4375-bccf-d2caff4f515e_isrForCrosstalkSources_1125907_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/7dd49a0a-1238-4375-bccf-d2caff4f515e_isrForCrosstalkSources_1125907_54.stderr parsl.dataflow.dflow DEBUG: Task 1564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1564 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1564 has memoization hash 55608e5095a8c109e6dc0e5cf88f0b1d parsl.dataflow.memoization INFO: Task 1564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1564 try 0 launched on executor multi with executor id 1565 parsl.dataflow.dflow INFO: Standard output for task 1564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/342f472f-52b2-4320-858d-34eab12d541c_isrForCrosstalkSources_1125895_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/342f472f-52b2-4320-858d-34eab12d541c_isrForCrosstalkSources_1125895_5.stderr parsl.dataflow.dflow DEBUG: Task 1565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1565 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1565 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1565 has memoization hash beebabc89e1b5ceea6def00ee905c3d3 parsl.dataflow.memoization INFO: Task 1565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1565 try 0 launched on executor multi with executor id 1566 parsl.dataflow.dflow INFO: Standard output for task 1565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/a819b7f0-e9ae-4d3e-929e-1b056c0d6311_isrForCrosstalkSources_1125916_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/a819b7f0-e9ae-4d3e-929e-1b056c0d6311_isrForCrosstalkSources_1125916_39.stderr parsl.dataflow.dflow DEBUG: Task 1566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1566 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1566 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1566 has memoization hash af7f4579a0fa862293c9f4da3a12ad71 parsl.dataflow.memoization INFO: Task 1566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a353a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1566 try 0 launched on executor multi with executor id 1567 parsl.dataflow.dflow INFO: Standard output for task 1566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/4376f5e2-88be-4980-8cf8-c9a5126d211a_isrForCrosstalkSources_1125904_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/4376f5e2-88be-4980-8cf8-c9a5126d211a_isrForCrosstalkSources_1125904_62.stderr parsl.dataflow.dflow DEBUG: Task 1567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1567 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1567 has memoization hash 716f4e1371ab2d9ba42d1a0b219228ba parsl.dataflow.memoization INFO: Task 1567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a354e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1567 try 0 launched on executor multi with executor id 1568 parsl.dataflow.dflow INFO: Standard output for task 1567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/27c07e48-bd12-4300-b570-ec71f59bf526_isrForCrosstalkSources_1125903_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/27c07e48-bd12-4300-b570-ec71f59bf526_isrForCrosstalkSources_1125903_32.stderr parsl.dataflow.dflow DEBUG: Task 1568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1568 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1568 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1568 has memoization hash e4feef8c802afc2e6c1d0a9a1506b91d parsl.dataflow.memoization INFO: Task 1568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1568 try 0 launched on executor multi with executor id 1569 parsl.dataflow.dflow INFO: Standard output for task 1568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/fc5b27e6-fb09-4a00-850e-c1e408acbcfb_isrForCrosstalkSources_1125898_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/fc5b27e6-fb09-4a00-850e-c1e408acbcfb_isrForCrosstalkSources_1125898_2.stderr parsl.dataflow.dflow DEBUG: Task 1569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1569 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1569 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1569 has memoization hash cfdea86abfdd7406aa60d0ec7479ee9e parsl.dataflow.memoization INFO: Task 1569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1569 try 0 launched on executor multi with executor id 1570 parsl.dataflow.dflow INFO: Standard output for task 1569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/136bcd04-7a39-4152-9f35-4e79cb96e2a6_isrForCrosstalkSources_1125939_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/136bcd04-7a39-4152-9f35-4e79cb96e2a6_isrForCrosstalkSources_1125939_35.stderr parsl.dataflow.dflow DEBUG: Task 1570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1570 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1570 has memoization hash b9b9178ac805f1925a7613b1ccfb41fb parsl.dataflow.memoization INFO: Task 1570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1570 try 0 launched on executor multi with executor id 1571 parsl.dataflow.dflow INFO: Standard output for task 1570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e3e931af-468e-4073-9728-6a07e9fca96b_isrForCrosstalkSources_1125895_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e3e931af-468e-4073-9728-6a07e9fca96b_isrForCrosstalkSources_1125895_9.stderr parsl.dataflow.dflow DEBUG: Task 1571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1571 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1571 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1571 has memoization hash e50569a6ce14181650b83963921561d1 parsl.dataflow.memoization INFO: Task 1571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a356c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1571 try 0 launched on executor multi with executor id 1572 parsl.dataflow.dflow INFO: Standard output for task 1571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/fdccedcf-d4d8-4b8f-843a-42c48bbbd085_isrForCrosstalkSources_1125928_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/fdccedcf-d4d8-4b8f-843a-42c48bbbd085_isrForCrosstalkSources_1125928_26.stderr parsl.dataflow.dflow DEBUG: Task 1572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1572 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1572 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1572 has memoization hash 5917c46681daa5db53a79f489c9e59d5 parsl.dataflow.memoization INFO: Task 1572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1572 try 0 launched on executor multi with executor id 1573 parsl.dataflow.dflow INFO: Standard output for task 1572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/e19325cb-9d67-4099-b681-b8b0d429083e_isrForCrosstalkSources_1125907_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/e19325cb-9d67-4099-b681-b8b0d429083e_isrForCrosstalkSources_1125907_7.stderr parsl.dataflow.dflow DEBUG: Task 1573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1573 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1573 has memoization hash eb4132d12a99fc979afd0d55c6c84c3d parsl.dataflow.memoization INFO: Task 1573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1573 try 0 launched on executor multi with executor id 1574 parsl.dataflow.dflow INFO: Standard output for task 1573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/60340e23-af89-4a70-af9b-ad280bbcd9a7_isrForCrosstalkSources_1125909_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/60340e23-af89-4a70-af9b-ad280bbcd9a7_isrForCrosstalkSources_1125909_6.stderr parsl.dataflow.dflow DEBUG: Task 1574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1574 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1574 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1574 has memoization hash 9b9cec1a7a88b695fe2642b859d77aab parsl.dataflow.memoization INFO: Task 1574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a358a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1574 try 0 launched on executor multi with executor id 1575 parsl.dataflow.dflow INFO: Standard output for task 1574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/0089ef97-5f42-4d62-8162-c25ae7306234_isrForCrosstalkSources_1125888_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/0089ef97-5f42-4d62-8162-c25ae7306234_isrForCrosstalkSources_1125888_3.stderr parsl.dataflow.dflow DEBUG: Task 1575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1575 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1575 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1575 has memoization hash 79a9645a17d18b3ed010423978fc438d parsl.dataflow.memoization INFO: Task 1575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1575 try 0 launched on executor multi with executor id 1576 parsl.dataflow.dflow INFO: Standard output for task 1575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/9e89495d-ec54-4bc8-91d8-783b18974edf_isrForCrosstalkSources_1125928_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/9e89495d-ec54-4bc8-91d8-783b18974edf_isrForCrosstalkSources_1125928_19.stderr parsl.dataflow.dflow DEBUG: Task 1576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1576 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1576 has memoization hash 267619e936bddbd7ff4ff0e37f6377e1 parsl.dataflow.memoization INFO: Task 1576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a359e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1576 try 0 launched on executor multi with executor id 1577 parsl.dataflow.dflow INFO: Standard output for task 1576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/26fa1884-7bdf-4e95-9eb9-176da691d89c_isrForCrosstalkSources_1125904_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/26fa1884-7bdf-4e95-9eb9-176da691d89c_isrForCrosstalkSources_1125904_26.stderr parsl.dataflow.dflow DEBUG: Task 1577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1577 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1577 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1577 has memoization hash 78aa781ac17fc70e4e9a7af29dfa9adf parsl.dataflow.memoization INFO: Task 1577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1577 try 0 launched on executor multi with executor id 1578 parsl.dataflow.dflow INFO: Standard output for task 1577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/1fce1d9a-1224-4dcc-b9b4-ba18585ddeeb_isrForCrosstalkSources_1125936_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/1fce1d9a-1224-4dcc-b9b4-ba18585ddeeb_isrForCrosstalkSources_1125936_32.stderr parsl.dataflow.dflow DEBUG: Task 1578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1578 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1578 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1578 has memoization hash ffefd6eae6215386a96caa4a59bed82f parsl.dataflow.memoization INFO: Task 1578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1578 try 0 launched on executor multi with executor id 1579 parsl.dataflow.dflow INFO: Standard output for task 1578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/0d419855-baf6-4c77-939f-ade1fc5d699b_isrForCrosstalkSources_1125936_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/0d419855-baf6-4c77-939f-ade1fc5d699b_isrForCrosstalkSources_1125936_41.stderr parsl.dataflow.dflow DEBUG: Task 1579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1579 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1579 has memoization hash a72e40fd2e596a3ebe4cc4aea73aab82 parsl.dataflow.memoization INFO: Task 1579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1579 try 0 launched on executor multi with executor id 1580 parsl.dataflow.dflow INFO: Standard output for task 1579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/56c066d2-e8af-41fe-809e-e0176855bd52_isrForCrosstalkSources_1125934_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/56c066d2-e8af-41fe-809e-e0176855bd52_isrForCrosstalkSources_1125934_6.stderr parsl.dataflow.dflow DEBUG: Task 1580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1580 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1580 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1580 has memoization hash 69332bde33fc9adc3b2498bdb536d37f parsl.dataflow.memoization INFO: Task 1580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1580 try 0 launched on executor multi with executor id 1581 parsl.dataflow.dflow INFO: Standard output for task 1580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/df1bf58a-323e-420a-8932-5484b0c39cd0_isrForCrosstalkSources_1125945_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/df1bf58a-323e-420a-8932-5484b0c39cd0_isrForCrosstalkSources_1125945_13.stderr parsl.dataflow.dflow DEBUG: Task 1581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1581 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1581 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1581 has memoization hash ae0f182f2ae7dd00c7938e1e0a0d6ade parsl.dataflow.memoization INFO: Task 1581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1581 try 0 launched on executor multi with executor id 1582 parsl.dataflow.dflow INFO: Standard output for task 1581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/ffce5a6c-6794-4e6a-b6d8-4d1b1fd079c9_isrForCrosstalkSources_1125935_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/ffce5a6c-6794-4e6a-b6d8-4d1b1fd079c9_isrForCrosstalkSources_1125935_58.stderr parsl.dataflow.dflow DEBUG: Task 1582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1582 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1582 has memoization hash 08e69473a9abb07ab89756c30ae3a5d3 parsl.dataflow.memoization INFO: Task 1582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1582 try 0 launched on executor multi with executor id 1583 parsl.dataflow.dflow INFO: Standard output for task 1582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/0fac113c-4c05-47c3-a247-ed8aec0a6f3e_isrForCrosstalkSources_1125918_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/0fac113c-4c05-47c3-a247-ed8aec0a6f3e_isrForCrosstalkSources_1125918_8.stderr parsl.dataflow.dflow DEBUG: Task 1583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1583 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1583 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1583 has memoization hash 33d9881c3cbdc38013dac3886aaa95d3 parsl.dataflow.memoization INFO: Task 1583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1583 try 0 launched on executor multi with executor id 1584 parsl.dataflow.dflow INFO: Standard output for task 1583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/06e5997a-52ce-469b-9adb-97680884b881_isrForCrosstalkSources_1125923_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/06e5997a-52ce-469b-9adb-97680884b881_isrForCrosstalkSources_1125923_11.stderr parsl.dataflow.dflow DEBUG: Task 1584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1584 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1584 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1584 has memoization hash 01d771010e7a9c93677c5e004848620a parsl.dataflow.memoization INFO: Task 1584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1584 try 0 launched on executor multi with executor id 1585 parsl.dataflow.dflow INFO: Standard output for task 1584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/b71db0f5-6e2a-4f1b-bcb3-c1fa7e9fd455_isrForCrosstalkSources_1125924_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/b71db0f5-6e2a-4f1b-bcb3-c1fa7e9fd455_isrForCrosstalkSources_1125924_14.stderr parsl.dataflow.dflow DEBUG: Task 1585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1585 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1585 has memoization hash 709ef32f3c7ac6bf35bf4cd3786a532e parsl.dataflow.memoization INFO: Task 1585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a35f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1585 try 0 launched on executor multi with executor id 1586 parsl.dataflow.dflow INFO: Standard output for task 1585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/fb9be19a-0de2-42d2-b4d2-782caec1d68f_isrForCrosstalkSources_1125908_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/fb9be19a-0de2-42d2-b4d2-782caec1d68f_isrForCrosstalkSources_1125908_52.stderr parsl.dataflow.dflow DEBUG: Task 1586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1586 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1586 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1586 has memoization hash a6e7c40620eb093b976ed597194c1a11 parsl.dataflow.memoization INFO: Task 1586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1586 try 0 launched on executor multi with executor id 1587 parsl.dataflow.dflow INFO: Standard output for task 1586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/3975b170-e48c-403a-9b46-d93783c44b81_isrForCrosstalkSources_1125916_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/3975b170-e48c-403a-9b46-d93783c44b81_isrForCrosstalkSources_1125916_45.stderr parsl.dataflow.dflow DEBUG: Task 1587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1587 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1587 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1587 has memoization hash 74b9fbbab09800df8676d583e18d8c56 parsl.dataflow.memoization INFO: Task 1587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1587 try 0 launched on executor multi with executor id 1588 parsl.dataflow.dflow INFO: Standard output for task 1587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/bd468e3e-f352-43fa-a57d-d10fc2348d65_isrForCrosstalkSources_1125915_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/bd468e3e-f352-43fa-a57d-d10fc2348d65_isrForCrosstalkSources_1125915_46.stderr parsl.dataflow.dflow DEBUG: Task 1588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1588 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1588 has memoization hash 3c1cc4e5a5d467f4dd908dc1a79a9216 parsl.dataflow.memoization INFO: Task 1588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a360c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1588 try 0 launched on executor multi with executor id 1589 parsl.dataflow.dflow INFO: Standard output for task 1588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/64789d49-7370-4bc2-b302-8ac5bda30f3c_isrForCrosstalkSources_1125924_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/64789d49-7370-4bc2-b302-8ac5bda30f3c_isrForCrosstalkSources_1125924_48.stderr parsl.dataflow.dflow DEBUG: Task 1589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1589 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1589 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1589 has memoization hash 930d650f6bde1139033b033bf81addc0 parsl.dataflow.memoization INFO: Task 1589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1589 try 0 launched on executor multi with executor id 1590 parsl.dataflow.dflow INFO: Standard output for task 1589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/6a063722-59dd-44a8-818c-1bda4f44a811_isrForCrosstalkSources_1125889_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/6a063722-59dd-44a8-818c-1bda4f44a811_isrForCrosstalkSources_1125889_16.stderr parsl.dataflow.dflow DEBUG: Task 1590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1590 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1590 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1590 has memoization hash a250e009d82f8bec9f4c86f467540e4b parsl.dataflow.memoization INFO: Task 1590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a362a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1590 try 0 launched on executor multi with executor id 1591 parsl.dataflow.dflow INFO: Standard output for task 1590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/e6b2d57f-5711-4ae6-8f66-8890c95f9604_isrForCrosstalkSources_1125896_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/e6b2d57f-5711-4ae6-8f66-8890c95f9604_isrForCrosstalkSources_1125896_1.stderr parsl.dataflow.dflow DEBUG: Task 1591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1591 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1591 has memoization hash a5289a1351e34be41d3223cdb6c6635c parsl.dataflow.memoization INFO: Task 1591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1591 try 0 launched on executor multi with executor id 1592 parsl.dataflow.dflow INFO: Standard output for task 1591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/14ec24fd-3aee-471d-8622-ffcdec1c9481_isrForCrosstalkSources_1125891_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/14ec24fd-3aee-471d-8622-ffcdec1c9481_isrForCrosstalkSources_1125891_39.stderr parsl.dataflow.dflow DEBUG: Task 1592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1592 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1592 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1592 has memoization hash d91ed9e6d91ac92bfd3763eac69816fc parsl.dataflow.memoization INFO: Task 1592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1592 try 0 launched on executor multi with executor id 1593 parsl.dataflow.dflow INFO: Standard output for task 1592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/38aa3d9c-2bb1-49cf-be8a-5823791a7ee7_isrForCrosstalkSources_1125891_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/38aa3d9c-2bb1-49cf-be8a-5823791a7ee7_isrForCrosstalkSources_1125891_27.stderr parsl.dataflow.dflow DEBUG: Task 1593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1593 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1593 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1593 has memoization hash d760424a20c76929dcbea0d8382e60dd parsl.dataflow.memoization INFO: Task 1593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a363e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1593 try 0 launched on executor multi with executor id 1594 parsl.dataflow.dflow INFO: Standard output for task 1593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/305bc320-68b9-486a-9a08-b9835130aa15_isrForCrosstalkSources_1125941_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/305bc320-68b9-486a-9a08-b9835130aa15_isrForCrosstalkSources_1125941_18.stderr parsl.dataflow.dflow DEBUG: Task 1594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1594 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1594 has memoization hash c0c34eb408a1d51e0e87fc5e6739fcad parsl.dataflow.memoization INFO: Task 1594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1594 try 0 launched on executor multi with executor id 1595 parsl.dataflow.dflow INFO: Standard output for task 1594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/f4ca7170-bee1-434b-8e7f-7dc88cc4b2dd_isrForCrosstalkSources_1125914_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/f4ca7170-bee1-434b-8e7f-7dc88cc4b2dd_isrForCrosstalkSources_1125914_55.stderr parsl.dataflow.dflow DEBUG: Task 1595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1595 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1595 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1595 has memoization hash 23400902371cc7d32b1e1a8e835c71f7 parsl.dataflow.memoization INFO: Task 1595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a365c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1595 try 0 launched on executor multi with executor id 1596 parsl.dataflow.dflow INFO: Standard output for task 1595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0d36ae6d-5f91-4983-9af6-4470c845e157_isrForCrosstalkSources_1125890_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0d36ae6d-5f91-4983-9af6-4470c845e157_isrForCrosstalkSources_1125890_45.stderr parsl.dataflow.dflow DEBUG: Task 1596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1596 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1596 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1596 has memoization hash 2c5aa98cc1e3e894893f18fbfaeadc05 parsl.dataflow.memoization INFO: Task 1596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1596 try 0 launched on executor multi with executor id 1597 parsl.dataflow.dflow INFO: Standard output for task 1596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3a519562-726a-4a35-b61a-6a35fdd2fbbc_isrForCrosstalkSources_1125912_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3a519562-726a-4a35-b61a-6a35fdd2fbbc_isrForCrosstalkSources_1125912_17.stderr parsl.dataflow.dflow DEBUG: Task 1597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1597 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1597 has memoization hash a036937d8679db63197ca4e056ba0e4b parsl.dataflow.memoization INFO: Task 1597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a367a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1597 try 0 launched on executor multi with executor id 1598 parsl.dataflow.dflow INFO: Standard output for task 1597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/9c360ed8-88c5-428c-9cca-c7ef432118eb_isrForCrosstalkSources_1125932_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/9c360ed8-88c5-428c-9cca-c7ef432118eb_isrForCrosstalkSources_1125932_27.stderr parsl.dataflow.dflow DEBUG: Task 1598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1598 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1598 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1598 has memoization hash fc0c8016e0ef7f4bf4cf204d83447d5f parsl.dataflow.memoization INFO: Task 1598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1598 try 0 launched on executor multi with executor id 1599 parsl.dataflow.dflow INFO: Standard output for task 1598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/f6182662-3c66-4c3e-90db-0613379b2b47_isrForCrosstalkSources_1125921_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/f6182662-3c66-4c3e-90db-0613379b2b47_isrForCrosstalkSources_1125921_1.stderr parsl.dataflow.dflow DEBUG: Task 1599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1599 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1599 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1599 has memoization hash 809697c4081435fdec675ee16e25104e parsl.dataflow.memoization INFO: Task 1599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:51 proc_lsst.multi:170 [INFO] [multi] job_id 16896414 proc_lsst.multi INFO: [multi] job_id 16896414 2024-03-07 15:06:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 15:06:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896414 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896414 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16896414 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-07 15:06:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-07 15:06:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 15:06:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1599 try 0 launched on executor multi with executor id 1600 parsl.dataflow.dflow INFO: Standard output for task 1599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/47eeb897-15b7-40d5-a0ea-91e117c386ba_isrForCrosstalkSources_1125921_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/47eeb897-15b7-40d5-a0ea-91e117c386ba_isrForCrosstalkSources_1125921_61.stderr parsl.dataflow.dflow DEBUG: Task 1600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1600 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1600 has memoization hash a0788d91c75e7d8b5c722e73903ccb4a parsl.dataflow.memoization INFO: Task 1600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1600 try 0 launched on executor multi with executor id 1601 parsl.dataflow.dflow INFO: Standard output for task 1600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/d9a2f24c-41ff-4cf1-9640-af5ba747c5b9_isrForCrosstalkSources_1125908_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/d9a2f24c-41ff-4cf1-9640-af5ba747c5b9_isrForCrosstalkSources_1125908_18.stderr parsl.dataflow.dflow DEBUG: Task 1601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1601 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1601 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1601 has memoization hash 0217397fc200c83343520967965cea7b parsl.dataflow.memoization INFO: Task 1601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1601 try 0 launched on executor multi with executor id 1602 parsl.dataflow.dflow INFO: Standard output for task 1601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/c99f87ed-d265-4f73-8cbb-e48c9aea31ac_isrForCrosstalkSources_1125918_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/c99f87ed-d265-4f73-8cbb-e48c9aea31ac_isrForCrosstalkSources_1125918_57.stderr parsl.dataflow.dflow DEBUG: Task 1602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1602 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1602 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1602 has memoization hash 954fb0f5bc28b982f7abe39e4d625c5a parsl.dataflow.memoization INFO: Task 1602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1602 try 0 launched on executor multi with executor id 1603 parsl.dataflow.dflow INFO: Standard output for task 1602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/eeb98cd7-a14a-43ce-903f-917c3cc7cece_isrForCrosstalkSources_1125905_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/eeb98cd7-a14a-43ce-903f-917c3cc7cece_isrForCrosstalkSources_1125905_29.stderr parsl.dataflow.dflow DEBUG: Task 1603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1603 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1603 has memoization hash 0a53455a4ae6197182e89901c6dc2295 parsl.dataflow.memoization INFO: Task 1603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1603 try 0 launched on executor multi with executor id 1604 parsl.dataflow.dflow INFO: Standard output for task 1603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/f557620c-15d2-4289-a207-dc42a45e18f4_isrForCrosstalkSources_1125919_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/f557620c-15d2-4289-a207-dc42a45e18f4_isrForCrosstalkSources_1125919_46.stderr parsl.dataflow.dflow DEBUG: Task 1604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1604 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1604 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1604 has memoization hash 196f433c5fdc6cb58e4d9e929d9f2727 parsl.dataflow.memoization INFO: Task 1604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a34040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1604 try 0 launched on executor multi with executor id 1605 parsl.dataflow.dflow INFO: Standard output for task 1604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/f5a3028a-a158-42f5-96c7-7fa1cb335596_isrForCrosstalkSources_1125899_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/f5a3028a-a158-42f5-96c7-7fa1cb335596_isrForCrosstalkSources_1125899_12.stderr parsl.dataflow.dflow DEBUG: Task 1605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1605 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1605 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1605 has memoization hash 044462c79ab53ccd5a7b777f831896e7 parsl.dataflow.memoization INFO: Task 1605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1605 try 0 launched on executor multi with executor id 1606 parsl.dataflow.dflow INFO: Standard output for task 1605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/2eb7a2f4-c031-4c7c-9f89-d4aa4c682990_isrForCrosstalkSources_1125914_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/2eb7a2f4-c031-4c7c-9f89-d4aa4c682990_isrForCrosstalkSources_1125914_53.stderr parsl.dataflow.dflow DEBUG: Task 1606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1606 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1606 has memoization hash 2258126d2922c94fb74465bcf1d6f6ad parsl.dataflow.memoization INFO: Task 1606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1606 try 0 launched on executor multi with executor id 1607 parsl.dataflow.dflow INFO: Standard output for task 1606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2016704d-3ef9-43de-8a97-7f7dac8d2e84_isrForCrosstalkSources_1125933_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2016704d-3ef9-43de-8a97-7f7dac8d2e84_isrForCrosstalkSources_1125933_12.stderr parsl.dataflow.dflow DEBUG: Task 1607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1607 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1607 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1607 has memoization hash fe39f56343c5a6bb1add77173c85267a parsl.dataflow.memoization INFO: Task 1607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1607 try 0 launched on executor multi with executor id 1608 parsl.dataflow.dflow INFO: Standard output for task 1607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/50bf3edf-db6f-44a1-9f7b-c2184fe1c862_isrForCrosstalkSources_1125908_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/50bf3edf-db6f-44a1-9f7b-c2184fe1c862_isrForCrosstalkSources_1125908_54.stderr parsl.dataflow.dflow DEBUG: Task 1608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1608 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1608 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1608 has memoization hash 42d14bc139dc2708e5d36a35c958a1ab parsl.dataflow.memoization INFO: Task 1608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1608 try 0 launched on executor multi with executor id 1609 parsl.dataflow.dflow INFO: Standard output for task 1608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/282f3114-9bd8-4a47-8190-c917da5cdb3e_isrForCrosstalkSources_1125938_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/282f3114-9bd8-4a47-8190-c917da5cdb3e_isrForCrosstalkSources_1125938_34.stderr parsl.dataflow.dflow DEBUG: Task 1609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1609 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1609 has memoization hash e2609c978f30e8d5b4e3b33c9b19abd5 parsl.dataflow.memoization INFO: Task 1609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1609 try 0 launched on executor multi with executor id 1610 parsl.dataflow.dflow INFO: Standard output for task 1609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/1647f9a0-d53e-4e36-8035-1e52b9611bca_isrForCrosstalkSources_1125935_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/1647f9a0-d53e-4e36-8035-1e52b9611bca_isrForCrosstalkSources_1125935_49.stderr parsl.dataflow.dflow DEBUG: Task 1610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1610 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1610 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1610 has memoization hash 9b8f5c3ef4a8e27ad060535851446af3 parsl.dataflow.memoization INFO: Task 1610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1610 try 0 launched on executor multi with executor id 1611 parsl.dataflow.dflow INFO: Standard output for task 1610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ddad1ffd-ba75-4c1d-8783-79b3db196c64_isrForCrosstalkSources_1125947_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ddad1ffd-ba75-4c1d-8783-79b3db196c64_isrForCrosstalkSources_1125947_11.stderr parsl.dataflow.dflow DEBUG: Task 1611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1611 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1611 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1611 has memoization hash cb3029d04b96f2b71168e10c7328fd1b parsl.dataflow.memoization INFO: Task 1611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1611 try 0 launched on executor multi with executor id 1612 parsl.dataflow.dflow INFO: Standard output for task 1611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/06985c0d-d682-487f-a4e2-a235a0e3298d_isrForCrosstalkSources_1125943_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/06985c0d-d682-487f-a4e2-a235a0e3298d_isrForCrosstalkSources_1125943_51.stderr parsl.dataflow.dflow DEBUG: Task 1612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1612 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1612 has memoization hash 2a62b5a3c6f43b6851c893bc6ed69d96 parsl.dataflow.memoization INFO: Task 1612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1612 try 0 launched on executor multi with executor id 1613 parsl.dataflow.dflow INFO: Standard output for task 1612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/38d656f5-ae1f-4e4d-8106-11c938553b1d_isrForCrosstalkSources_1125888_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/38d656f5-ae1f-4e4d-8106-11c938553b1d_isrForCrosstalkSources_1125888_40.stderr parsl.dataflow.dflow DEBUG: Task 1613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1613 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1613 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1613 has memoization hash fcc2e02c4bc809206953b0122a7c1218 parsl.dataflow.memoization INFO: Task 1613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1613 try 0 launched on executor multi with executor id 1614 parsl.dataflow.dflow INFO: Standard output for task 1613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/32e44fb8-aa84-4645-aefb-691397c85b5f_isrForCrosstalkSources_1125901_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/32e44fb8-aa84-4645-aefb-691397c85b5f_isrForCrosstalkSources_1125901_28.stderr parsl.dataflow.dflow DEBUG: Task 1614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1614 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1614 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1614 has memoization hash 078e790095ced56e7878be7f23ccfe5c parsl.dataflow.memoization INFO: Task 1614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a371a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1614 try 0 launched on executor multi with executor id 1615 parsl.dataflow.dflow INFO: Standard output for task 1614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/d26eebc4-2a06-44e5-895f-596d0618a63b_isrForCrosstalkSources_1125918_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/d26eebc4-2a06-44e5-895f-596d0618a63b_isrForCrosstalkSources_1125918_10.stderr parsl.dataflow.dflow DEBUG: Task 1615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1615 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1615 has memoization hash ee65f1c1cf7468dfaaf064f2fc50cd22 parsl.dataflow.memoization INFO: Task 1615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1615 try 0 launched on executor multi with executor id 1616 parsl.dataflow.dflow INFO: Standard output for task 1615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/26f88e4d-f85a-48ba-aca7-1a499e9727bd_isrForCrosstalkSources_1125897_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/26f88e4d-f85a-48ba-aca7-1a499e9727bd_isrForCrosstalkSources_1125897_16.stderr parsl.dataflow.dflow DEBUG: Task 1616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1616 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1616 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1616 has memoization hash 59ebe2e38b459ac648a6408eb31c013f parsl.dataflow.memoization INFO: Task 1616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1616 try 0 launched on executor multi with executor id 1617 parsl.dataflow.dflow INFO: Standard output for task 1616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/e1996f19-a4e6-4781-a794-6277d125b156_isrForCrosstalkSources_1125931_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/e1996f19-a4e6-4781-a794-6277d125b156_isrForCrosstalkSources_1125931_6.stderr parsl.dataflow.dflow DEBUG: Task 1617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1617 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1617 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1617 has memoization hash 81e1434cd1d40ef84ff8a7e1c282ffc6 parsl.dataflow.memoization INFO: Task 1617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a372e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1617 try 0 launched on executor multi with executor id 1618 parsl.dataflow.dflow INFO: Standard output for task 1617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/151f9ead-a4b7-4ea4-8e95-751e49edd876_isrForCrosstalkSources_1125895_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/151f9ead-a4b7-4ea4-8e95-751e49edd876_isrForCrosstalkSources_1125895_29.stderr parsl.dataflow.dflow DEBUG: Task 1618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1618 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1618 has memoization hash 0129acfec9e1dc2593a24e01651f5f06 parsl.dataflow.memoization INFO: Task 1618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1618 try 0 launched on executor multi with executor id 1619 parsl.dataflow.dflow INFO: Standard output for task 1618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/798f8671-acb8-41cd-b9f2-2f63ec787f16_isrForCrosstalkSources_1125943_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/798f8671-acb8-41cd-b9f2-2f63ec787f16_isrForCrosstalkSources_1125943_14.stderr parsl.dataflow.dflow DEBUG: Task 1619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1619 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1619 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1619 has memoization hash 010d31d5325bf3008ed603e02db5d20a parsl.dataflow.memoization INFO: Task 1619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a374c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1619 try 0 launched on executor multi with executor id 1620 parsl.dataflow.dflow INFO: Standard output for task 1619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/db515f94-ffca-482d-8384-a3fc0d4d4c01_isrForCrosstalkSources_1125944_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/db515f94-ffca-482d-8384-a3fc0d4d4c01_isrForCrosstalkSources_1125944_10.stderr parsl.dataflow.dflow DEBUG: Task 1620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1620 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1620 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1620 has memoization hash 0c8637e86b03d43296a36d423ec52cdc parsl.dataflow.memoization INFO: Task 1620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1620 try 0 launched on executor multi with executor id 1621 parsl.dataflow.dflow INFO: Standard output for task 1620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/ea2fb698-ed5a-4cbc-9469-c9f152a7c78b_isrForCrosstalkSources_1125942_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/ea2fb698-ed5a-4cbc-9469-c9f152a7c78b_isrForCrosstalkSources_1125942_50.stderr parsl.dataflow.dflow DEBUG: Task 1621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1621 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1621 has memoization hash 6d22a36f26c20f52d3b165d8e605ba22 parsl.dataflow.memoization INFO: Task 1621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1621 try 0 launched on executor multi with executor id 1622 parsl.dataflow.dflow INFO: Standard output for task 1621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/c8b2d7d4-bce3-484a-aef7-0ffe47b38b24_isrForCrosstalkSources_1125935_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/c8b2d7d4-bce3-484a-aef7-0ffe47b38b24_isrForCrosstalkSources_1125935_7.stderr parsl.dataflow.dflow DEBUG: Task 1622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1622 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1622 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1622 has memoization hash fa8a9138cc68a7cdd1d20a3594a7f177 parsl.dataflow.memoization INFO: Task 1622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a377e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1622 try 0 launched on executor multi with executor id 1623 parsl.dataflow.dflow INFO: Standard output for task 1622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/793bba36-d31b-45c3-8b2e-f628c79ddf70_isrForCrosstalkSources_1125906_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/793bba36-d31b-45c3-8b2e-f628c79ddf70_isrForCrosstalkSources_1125906_4.stderr parsl.dataflow.dflow DEBUG: Task 1623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1623 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1623 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1623 has memoization hash 714fe1234900aa8bdb971101198f24b7 parsl.dataflow.memoization INFO: Task 1623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1623 try 0 launched on executor multi with executor id 1624 parsl.dataflow.dflow INFO: Standard output for task 1623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/3e9746ce-6f27-4c8a-88a8-ee8146576949_isrForCrosstalkSources_1125907_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/3e9746ce-6f27-4c8a-88a8-ee8146576949_isrForCrosstalkSources_1125907_12.stderr parsl.dataflow.dflow DEBUG: Task 1624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1624 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1624 has memoization hash f411ad29649a90ecf65947eff927de0f parsl.dataflow.memoization INFO: Task 1624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1624 try 0 launched on executor multi with executor id 1625 parsl.dataflow.dflow INFO: Standard output for task 1624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/1593a835-67b1-4ced-95b0-db5b578dc642_isrForCrosstalkSources_1125938_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/1593a835-67b1-4ced-95b0-db5b578dc642_isrForCrosstalkSources_1125938_19.stderr parsl.dataflow.dflow DEBUG: Task 1625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1625 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1625 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1625 has memoization hash c9bde24c921d58226ac941cb0d554706 parsl.dataflow.memoization INFO: Task 1625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a376a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1625 try 0 launched on executor multi with executor id 1626 parsl.dataflow.dflow INFO: Standard output for task 1625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/45ba2658-3f50-44bd-8ef3-3029d89ec7b8_isrForCrosstalkSources_1125922_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/45ba2658-3f50-44bd-8ef3-3029d89ec7b8_isrForCrosstalkSources_1125922_51.stderr parsl.dataflow.dflow DEBUG: Task 1626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1626 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1626 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1626 has memoization hash fe04a4a1192a4ede6cacc6ab9edb1995 parsl.dataflow.memoization INFO: Task 1626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1626 try 0 launched on executor multi with executor id 1627 parsl.dataflow.dflow INFO: Standard output for task 1626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/c2c53819-a327-4d38-84ae-9ce0504d23c4_isrForCrosstalkSources_1125907_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/c2c53819-a327-4d38-84ae-9ce0504d23c4_isrForCrosstalkSources_1125907_22.stderr parsl.dataflow.dflow DEBUG: Task 1627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1627 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1627 has memoization hash 92d19a7f9a69079f1626e5826d0732d8 parsl.dataflow.memoization INFO: Task 1627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1627 try 0 launched on executor multi with executor id 1628 parsl.dataflow.dflow INFO: Standard output for task 1627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/be55372b-f7c0-4d90-86ee-e990a3036106_isrForCrosstalkSources_1125925_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/be55372b-f7c0-4d90-86ee-e990a3036106_isrForCrosstalkSources_1125925_2.stderr parsl.dataflow.dflow DEBUG: Task 1628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1628 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1628 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1628 has memoization hash 2413a3a5e1c0282fd4fa1f76c1f42f8f parsl.dataflow.memoization INFO: Task 1628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1628 try 0 launched on executor multi with executor id 1629 parsl.dataflow.dflow INFO: Standard output for task 1628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/1ca024f6-397c-4128-b614-89c81a16446f_isrForCrosstalkSources_1125926_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/1ca024f6-397c-4128-b614-89c81a16446f_isrForCrosstalkSources_1125926_5.stderr parsl.dataflow.dflow DEBUG: Task 1629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1629 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1629 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1629 has memoization hash 71bd8619f2c8671ed7aa1bbe166a3e17 parsl.dataflow.memoization INFO: Task 1629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1629 try 0 launched on executor multi with executor id 1630 parsl.dataflow.dflow INFO: Standard output for task 1629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/6f67ea8e-0dfe-40b6-b4f0-ddc890caa7ec_isrForCrosstalkSources_1125936_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/6f67ea8e-0dfe-40b6-b4f0-ddc890caa7ec_isrForCrosstalkSources_1125936_58.stderr parsl.dataflow.dflow DEBUG: Task 1630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1630 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1630 has memoization hash b8ce89ff5586f2a05c777342fa9d82e6 parsl.dataflow.memoization INFO: Task 1630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a379c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1630 try 0 launched on executor multi with executor id 1631 parsl.dataflow.dflow INFO: Standard output for task 1630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/f9394f79-b06b-4b55-ab64-84f7d6de52d3_isrForCrosstalkSources_1125929_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/f9394f79-b06b-4b55-ab64-84f7d6de52d3_isrForCrosstalkSources_1125929_58.stderr parsl.dataflow.dflow DEBUG: Task 1631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1631 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1631 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1631 has memoization hash 14ebbc4c933908067701b857ea51637e parsl.dataflow.memoization INFO: Task 1631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1631 try 0 launched on executor multi with executor id 1632 parsl.dataflow.dflow INFO: Standard output for task 1631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ea3afd7e-cbe1-4bba-b2d1-5322e86e8fad_isrForCrosstalkSources_1125939_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ea3afd7e-cbe1-4bba-b2d1-5322e86e8fad_isrForCrosstalkSources_1125939_30.stderr parsl.dataflow.dflow DEBUG: Task 1632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1632 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1632 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1632 has memoization hash 119f084bbebbcc0b7c5585f6ebc586fe parsl.dataflow.memoization INFO: Task 1632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1632 try 0 launched on executor multi with executor id 1633 parsl.dataflow.dflow INFO: Standard output for task 1632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/5104b4e3-4891-4497-b1a0-8aead38a48ef_isrForCrosstalkSources_1125931_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/5104b4e3-4891-4497-b1a0-8aead38a48ef_isrForCrosstalkSources_1125931_24.stderr parsl.dataflow.dflow DEBUG: Task 1633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1633 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1633 has memoization hash 6da6d05c35ec4c4f9444ddd99df3b4ad parsl.dataflow.memoization INFO: Task 1633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1633 try 0 launched on executor multi with executor id 1634 parsl.dataflow.dflow INFO: Standard output for task 1633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/9724aa6f-0f4e-4a46-b523-a63d829efb66_isrForCrosstalkSources_1125920_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/9724aa6f-0f4e-4a46-b523-a63d829efb66_isrForCrosstalkSources_1125920_62.stderr parsl.dataflow.dflow DEBUG: Task 1634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1634 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1634 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1634 has memoization hash 468c0fa9f8d07a04b592b97cf3981f2f parsl.dataflow.memoization INFO: Task 1634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1634 try 0 launched on executor multi with executor id 1635 parsl.dataflow.dflow INFO: Standard output for task 1634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/9312b676-41c9-4c0f-81b0-cbacc53b0d21_isrForCrosstalkSources_1125902_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/9312b676-41c9-4c0f-81b0-cbacc53b0d21_isrForCrosstalkSources_1125902_10.stderr parsl.dataflow.dflow DEBUG: Task 1635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1635 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1635 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1635 has memoization hash 9191abaebeebe4693261b43997e64a31 parsl.dataflow.memoization INFO: Task 1635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1635 try 0 launched on executor multi with executor id 1636 parsl.dataflow.dflow INFO: Standard output for task 1635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/2b171272-0d4c-44bd-aa97-fe5bfb5687cb_isrForCrosstalkSources_1125894_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/2b171272-0d4c-44bd-aa97-fe5bfb5687cb_isrForCrosstalkSources_1125894_4.stderr parsl.dataflow.dflow DEBUG: Task 1636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1636 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1636 has memoization hash d9e05d12d327d91bb8f0808d70fba401 parsl.dataflow.memoization INFO: Task 1636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a37f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1636 try 0 launched on executor multi with executor id 1637 parsl.dataflow.dflow INFO: Standard output for task 1636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a36181da-4186-4288-9c17-6c73509b9956_isrForCrosstalkSources_1125921_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a36181da-4186-4288-9c17-6c73509b9956_isrForCrosstalkSources_1125921_37.stderr parsl.dataflow.dflow DEBUG: Task 1637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1637 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1637 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1637 has memoization hash 02910caf7b0437d0496bf149f0e0d0ee parsl.dataflow.memoization INFO: Task 1637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a368e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1637 try 0 launched on executor multi with executor id 1638 parsl.dataflow.dflow INFO: Standard output for task 1637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/61c48643-5150-4c2d-b00e-92a80a602c40_isrForCrosstalkSources_1125945_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/61c48643-5150-4c2d-b00e-92a80a602c40_isrForCrosstalkSources_1125945_3.stderr parsl.dataflow.dflow DEBUG: Task 1638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1638 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1638 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1638 has memoization hash 709307ead8a4821cc6273d6fd3efa392 parsl.dataflow.memoization INFO: Task 1638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1638 try 0 launched on executor multi with executor id 1639 parsl.dataflow.dflow INFO: Standard output for task 1638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/b955eaa0-9064-40a2-86ca-799af1182890_isrForCrosstalkSources_1125947_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/b955eaa0-9064-40a2-86ca-799af1182890_isrForCrosstalkSources_1125947_54.stderr parsl.dataflow.dflow DEBUG: Task 1639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1639 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1639 has memoization hash 534da39f69bbf5b2371a3357dc4ef911 parsl.dataflow.memoization INFO: Task 1639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1639 try 0 launched on executor multi with executor id 1640 parsl.dataflow.dflow INFO: Standard output for task 1639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/c5bf871b-009b-40af-99db-19e67e816833_isrForCrosstalkSources_1125944_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/c5bf871b-009b-40af-99db-19e67e816833_isrForCrosstalkSources_1125944_2.stderr parsl.dataflow.dflow DEBUG: Task 1640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1640 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1640 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1640 has memoization hash 320b3691e945f6a89d885a123669dd0a parsl.dataflow.memoization INFO: Task 1640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1640 try 0 launched on executor multi with executor id 1641 parsl.dataflow.dflow INFO: Standard output for task 1640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/179bd05d-9a53-489d-8015-867f235448f5_isrForCrosstalkSources_1125908_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/179bd05d-9a53-489d-8015-867f235448f5_isrForCrosstalkSources_1125908_47.stderr parsl.dataflow.dflow DEBUG: Task 1641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1641 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1641 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1641 has memoization hash d775df7970853eafe47e9634cf78b1f9 parsl.dataflow.memoization INFO: Task 1641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1641 try 0 launched on executor multi with executor id 1642 parsl.dataflow.dflow INFO: Standard output for task 1641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/f1ad5966-6720-4a3f-8ee4-ba194ee37bae_isrForCrosstalkSources_1125947_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/f1ad5966-6720-4a3f-8ee4-ba194ee37bae_isrForCrosstalkSources_1125947_60.stderr parsl.dataflow.dflow DEBUG: Task 1642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1642 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1642 has memoization hash 4f8afcf39699eac0f586e049dfcf8753 parsl.dataflow.memoization INFO: Task 1642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1642 try 0 launched on executor multi with executor id 1643 parsl.dataflow.dflow INFO: Standard output for task 1642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/b02b90b2-ef89-41a4-9052-9c56eb332099_isrForCrosstalkSources_1125901_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/b02b90b2-ef89-41a4-9052-9c56eb332099_isrForCrosstalkSources_1125901_33.stderr parsl.dataflow.dflow DEBUG: Task 1643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1643 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1643 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1643 has memoization hash 2e8171f82825e2044a6f476ec2c55aec parsl.dataflow.memoization INFO: Task 1643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1643 try 0 launched on executor multi with executor id 1644 parsl.dataflow.dflow INFO: Standard output for task 1643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/6a78237c-2765-4df4-b775-1497e9621e29_isrForCrosstalkSources_1125932_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/6a78237c-2765-4df4-b775-1497e9621e29_isrForCrosstalkSources_1125932_33.stderr parsl.dataflow.dflow DEBUG: Task 1644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1644 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1644 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1644 has memoization hash e720b850a9fe9fbec23569b58c0ec496 parsl.dataflow.memoization INFO: Task 1644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1644 try 0 launched on executor multi with executor id 1645 parsl.dataflow.dflow INFO: Standard output for task 1644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/10b3e21f-a006-416f-a08f-ddcb8de5a26a_isrForCrosstalkSources_1125889_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/10b3e21f-a006-416f-a08f-ddcb8de5a26a_isrForCrosstalkSources_1125889_7.stderr parsl.dataflow.dflow DEBUG: Task 1645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1645 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1645 has memoization hash 410b0b000835e0d4b0467a3d7bedf711 parsl.dataflow.memoization INFO: Task 1645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1645 try 0 launched on executor multi with executor id 1646 parsl.dataflow.dflow INFO: Standard output for task 1645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/a21bb7ce-f37f-4342-b211-6ebeaccfb3ac_isrForCrosstalkSources_1125920_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/a21bb7ce-f37f-4342-b211-6ebeaccfb3ac_isrForCrosstalkSources_1125920_7.stderr parsl.dataflow.dflow DEBUG: Task 1646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1646 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1646 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1646 has memoization hash 31cc7a4bbbd4b3bb5f6e0cdf0a01602f parsl.dataflow.memoization INFO: Task 1646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1646 try 0 launched on executor multi with executor id 1647 parsl.dataflow.dflow INFO: Standard output for task 1646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/f9000d28-47a1-42dc-a0d1-2ee1206ff6ca_isrForCrosstalkSources_1125895_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/f9000d28-47a1-42dc-a0d1-2ee1206ff6ca_isrForCrosstalkSources_1125895_14.stderr parsl.dataflow.dflow DEBUG: Task 1647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1647 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1647 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1647 has memoization hash a5535a5d9828ae8704c11da0c65a85d0 parsl.dataflow.memoization INFO: Task 1647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1647 try 0 launched on executor multi with executor id 1648 parsl.dataflow.dflow INFO: Standard output for task 1647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8cc53f6c-27e9-44d7-8d66-8a4a6257a716_isrForCrosstalkSources_1125891_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8cc53f6c-27e9-44d7-8d66-8a4a6257a716_isrForCrosstalkSources_1125891_16.stderr parsl.dataflow.dflow DEBUG: Task 1648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1648 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1648 has memoization hash e80f31b35f7185be850534270130848e parsl.dataflow.memoization INFO: Task 1648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1648 try 0 launched on executor multi with executor id 1649 parsl.dataflow.dflow INFO: Standard output for task 1648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/6eff3af2-1c6f-4aa1-b780-3e7c7cac58ce_isrForCrosstalkSources_1125910_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/6eff3af2-1c6f-4aa1-b780-3e7c7cac58ce_isrForCrosstalkSources_1125910_25.stderr parsl.dataflow.dflow DEBUG: Task 1649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1649 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1649 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1649 has memoization hash 47126316c51afbd30ab6a86adc5b5703 parsl.dataflow.memoization INFO: Task 1649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1649 try 0 launched on executor multi with executor id 1650 parsl.dataflow.dflow INFO: Standard output for task 1649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/5c3c1209-9698-42f2-b4f9-8e0911c1164c_isrForCrosstalkSources_1125943_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/5c3c1209-9698-42f2-b4f9-8e0911c1164c_isrForCrosstalkSources_1125943_10.stderr parsl.dataflow.dflow DEBUG: Task 1650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1650 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1650 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1650 has memoization hash fd537fec66c7e7d79c68d9890d505c29 parsl.dataflow.memoization INFO: Task 1650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1650 try 0 launched on executor multi with executor id 1651 parsl.dataflow.dflow INFO: Standard output for task 1650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/88602e0f-db3a-4ba5-8335-8bfc2d12d0ab_isrForCrosstalkSources_1125932_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/88602e0f-db3a-4ba5-8335-8bfc2d12d0ab_isrForCrosstalkSources_1125932_60.stderr parsl.dataflow.dflow DEBUG: Task 1651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1651 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1651 has memoization hash fe3e95d3ea75224f590334cbda0b3eca parsl.dataflow.memoization INFO: Task 1651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1651 try 0 launched on executor multi with executor id 1652 parsl.dataflow.dflow INFO: Standard output for task 1651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/0f987848-8860-4831-9c03-0051e22e4c6a_isrForCrosstalkSources_1125906_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/0f987848-8860-4831-9c03-0051e22e4c6a_isrForCrosstalkSources_1125906_56.stderr parsl.dataflow.dflow DEBUG: Task 1652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1652 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1652 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1652 has memoization hash c112cfd01ddae0bbb97588708677b243 parsl.dataflow.memoization INFO: Task 1652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1652 try 0 launched on executor multi with executor id 1653 parsl.dataflow.dflow INFO: Standard output for task 1652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/edf09f1e-f018-40b9-bf3f-0deea3cc87f7_isrForCrosstalkSources_1125932_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/edf09f1e-f018-40b9-bf3f-0deea3cc87f7_isrForCrosstalkSources_1125932_54.stderr parsl.dataflow.dflow DEBUG: Task 1653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1653 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1653 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1653 has memoization hash 249dae8ffcac75bad25cc89b762e9222 parsl.dataflow.memoization INFO: Task 1653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1653 try 0 launched on executor multi with executor id 1654 parsl.dataflow.dflow INFO: Standard output for task 1653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/6da3eacd-c970-4c87-a3e1-d539e806e9a9_isrForCrosstalkSources_1125903_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/6da3eacd-c970-4c87-a3e1-d539e806e9a9_isrForCrosstalkSources_1125903_35.stderr parsl.dataflow.dflow DEBUG: Task 1654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1654 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1654 has memoization hash 534a881027528a0926f2f07b1c1d9518 parsl.dataflow.memoization INFO: Task 1654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1654 try 0 launched on executor multi with executor id 1655 parsl.dataflow.dflow INFO: Standard output for task 1654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0c3d10a3-f6e0-4895-8ea6-fbbfaa598953_isrForCrosstalkSources_1125890_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0c3d10a3-f6e0-4895-8ea6-fbbfaa598953_isrForCrosstalkSources_1125890_43.stderr parsl.dataflow.dflow DEBUG: Task 1655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1655 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1655 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1655 has memoization hash 60740e14ad65498d8c253e4c74bd1ec3 parsl.dataflow.memoization INFO: Task 1655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1655 try 0 launched on executor multi with executor id 1656 parsl.dataflow.dflow INFO: Standard output for task 1655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/7f3a6a80-2711-4146-8a32-e2d985f383ac_isrForCrosstalkSources_1125892_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/7f3a6a80-2711-4146-8a32-e2d985f383ac_isrForCrosstalkSources_1125892_18.stderr parsl.dataflow.dflow DEBUG: Task 1656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1656 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1656 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1656 has memoization hash 0c95265e92e732679bfa6b2b334acaab parsl.dataflow.memoization INFO: Task 1656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1656 try 0 launched on executor multi with executor id 1657 parsl.dataflow.dflow INFO: Standard output for task 1656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/5b100163-0290-410e-9e8b-d26595a9a57a_isrForCrosstalkSources_1125918_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/5b100163-0290-410e-9e8b-d26595a9a57a_isrForCrosstalkSources_1125918_32.stderr parsl.dataflow.dflow DEBUG: Task 1657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1657 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1657 has memoization hash e8d07c755dae525cfa97b931bfd512ce parsl.dataflow.memoization INFO: Task 1657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1657 try 0 launched on executor multi with executor id 1658 parsl.dataflow.dflow INFO: Standard output for task 1657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a2380059-a33b-423b-8271-35238e54dc02_isrForCrosstalkSources_1125909_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a2380059-a33b-423b-8271-35238e54dc02_isrForCrosstalkSources_1125909_53.stderr parsl.dataflow.dflow DEBUG: Task 1658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1658 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1658 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1658 has memoization hash 74d91ed240359c100d19e714869cf9a1 parsl.dataflow.memoization INFO: Task 1658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1658 try 0 launched on executor multi with executor id 1659 parsl.dataflow.dflow INFO: Standard output for task 1658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/55476aba-9ef8-4746-9bd0-4e1181ba53c0_isrForCrosstalkSources_1125893_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/55476aba-9ef8-4746-9bd0-4e1181ba53c0_isrForCrosstalkSources_1125893_46.stderr parsl.dataflow.dflow DEBUG: Task 1659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1659 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1659 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1659 has memoization hash fe2bf648daf3b2489d2bf46b25cf1369 parsl.dataflow.memoization INFO: Task 1659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1659 try 0 launched on executor multi with executor id 1660 parsl.dataflow.dflow INFO: Standard output for task 1659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/7c57323f-1266-4647-9474-0574bca97bb2_isrForCrosstalkSources_1125911_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/7c57323f-1266-4647-9474-0574bca97bb2_isrForCrosstalkSources_1125911_46.stderr parsl.dataflow.dflow DEBUG: Task 1660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1660 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1660 has memoization hash ac52ec7215bf6cb1c5700f29aebccfd1 parsl.dataflow.memoization INFO: Task 1660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1660 try 0 launched on executor multi with executor id 1661 parsl.dataflow.dflow INFO: Standard output for task 1660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/d2b3c902-397e-4987-a382-38695feb219c_isrForCrosstalkSources_1125937_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/d2b3c902-397e-4987-a382-38695feb219c_isrForCrosstalkSources_1125937_39.stderr parsl.dataflow.dflow DEBUG: Task 1661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1661 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1661 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1661 has memoization hash 5304d14325c7ceaf99edd5a3224b5bd7 parsl.dataflow.memoization INFO: Task 1661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1661 try 0 launched on executor multi with executor id 1662 parsl.dataflow.dflow INFO: Standard output for task 1661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/f1c959c4-5e04-4bed-a41b-4b38d472ee64_isrForCrosstalkSources_1125934_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/f1c959c4-5e04-4bed-a41b-4b38d472ee64_isrForCrosstalkSources_1125934_3.stderr parsl.dataflow.dflow DEBUG: Task 1662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1662 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1662 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1662 has memoization hash 164fd3d6aa3e84ae13d944f3576b576f parsl.dataflow.memoization INFO: Task 1662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1662 try 0 launched on executor multi with executor id 1663 parsl.dataflow.dflow INFO: Standard output for task 1662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/f105db3a-c938-48ad-9c52-d45812a20867_isrForCrosstalkSources_1125911_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/f105db3a-c938-48ad-9c52-d45812a20867_isrForCrosstalkSources_1125911_18.stderr parsl.dataflow.dflow DEBUG: Task 1663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1663 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1663 has memoization hash c4379314cb9da33497caea4a6a195a73 parsl.dataflow.memoization INFO: Task 1663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1663 try 0 launched on executor multi with executor id 1664 parsl.dataflow.dflow INFO: Standard output for task 1663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/1d100df8-d656-46e6-b67a-1d68437136d5_isrForCrosstalkSources_1125910_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/1d100df8-d656-46e6-b67a-1d68437136d5_isrForCrosstalkSources_1125910_53.stderr parsl.dataflow.dflow DEBUG: Task 1664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1664 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1664 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1664 has memoization hash 112a2fc5e7bd6e4d537d8f1e0c919f1b parsl.dataflow.memoization INFO: Task 1664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1664 try 0 launched on executor multi with executor id 1665 parsl.dataflow.dflow INFO: Standard output for task 1664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/4e858019-6cdd-4766-a6a0-7c657fc152c8_isrForCrosstalkSources_1125907_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/4e858019-6cdd-4766-a6a0-7c657fc152c8_isrForCrosstalkSources_1125907_14.stderr parsl.dataflow.dflow DEBUG: Task 1665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1665 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1665 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1665 has memoization hash d76ce3e4e8f5994893fc6c38205b538e parsl.dataflow.memoization INFO: Task 1665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1665 try 0 launched on executor multi with executor id 1666 parsl.dataflow.dflow INFO: Standard output for task 1665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/60e643e4-9a5c-46dd-b447-6c524efe370d_isrForCrosstalkSources_1125942_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/60e643e4-9a5c-46dd-b447-6c524efe370d_isrForCrosstalkSources_1125942_41.stderr parsl.dataflow.dflow DEBUG: Task 1666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1666 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1666 has memoization hash 54502e3cf848d9a6fd36c20341a6b396 parsl.dataflow.memoization INFO: Task 1666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1666 try 0 launched on executor multi with executor id 1667 parsl.dataflow.dflow INFO: Standard output for task 1666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/fd403f33-81f3-487a-9b0e-99ddf329e9fa_isrForCrosstalkSources_1125931_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/fd403f33-81f3-487a-9b0e-99ddf329e9fa_isrForCrosstalkSources_1125931_53.stderr parsl.dataflow.dflow DEBUG: Task 1667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1667 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1667 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1667 has memoization hash bb26a1c641c766fb98e848af8557b3f6 parsl.dataflow.memoization INFO: Task 1667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1667 try 0 launched on executor multi with executor id 1668 parsl.dataflow.dflow INFO: Standard output for task 1667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c74d0ff8-fbd0-4684-8d5e-f73c75a339c2_isrForCrosstalkSources_1125904_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c74d0ff8-fbd0-4684-8d5e-f73c75a339c2_isrForCrosstalkSources_1125904_59.stderr parsl.dataflow.dflow DEBUG: Task 1668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1668 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1668 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1668 has memoization hash 0b340b4a801942ec1c23d56809d2139b parsl.dataflow.memoization INFO: Task 1668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1668 try 0 launched on executor multi with executor id 1669 parsl.dataflow.dflow INFO: Standard output for task 1668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/890ada41-287d-4f4a-8c9e-8844854493a3_isrForCrosstalkSources_1125914_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/890ada41-287d-4f4a-8c9e-8844854493a3_isrForCrosstalkSources_1125914_18.stderr parsl.dataflow.dflow DEBUG: Task 1669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1669 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1669 has memoization hash eea0b1d465362854a08c28b8e0e3f9e1 parsl.dataflow.memoization INFO: Task 1669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1669 try 0 launched on executor multi with executor id 1670 parsl.dataflow.dflow INFO: Standard output for task 1669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/2461f979-5745-4024-ab7f-a0708c310cec_isrForCrosstalkSources_1125910_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/2461f979-5745-4024-ab7f-a0708c310cec_isrForCrosstalkSources_1125910_2.stderr parsl.dataflow.dflow DEBUG: Task 1670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1670 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1670 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1670 has memoization hash d5a555b83d94de4cf28c30bd0365395c parsl.dataflow.memoization INFO: Task 1670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1670 try 0 launched on executor multi with executor id 1671 parsl.dataflow.dflow INFO: Standard output for task 1670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/dd4979d2-589e-4fbb-8853-9cd585cd100b_isrForCrosstalkSources_1125941_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/dd4979d2-589e-4fbb-8853-9cd585cd100b_isrForCrosstalkSources_1125941_44.stderr parsl.dataflow.dflow DEBUG: Task 1671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1671 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1671 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1671 has memoization hash cc72aaddcbdc560873bf340483baa945 parsl.dataflow.memoization INFO: Task 1671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1671 try 0 launched on executor multi with executor id 1672 parsl.dataflow.dflow INFO: Standard output for task 1671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/b4ab17da-d738-4c2a-88c0-c210b291eca0_isrForCrosstalkSources_1125925_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/b4ab17da-d738-4c2a-88c0-c210b291eca0_isrForCrosstalkSources_1125925_11.stderr parsl.dataflow.dflow DEBUG: Task 1672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1672 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1672 has memoization hash ba229739b008cbb7f169a99a386f35a6 parsl.dataflow.memoization INFO: Task 1672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1672 try 0 launched on executor multi with executor id 1673 parsl.dataflow.dflow INFO: Standard output for task 1672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/ce408f77-801f-42b3-b9eb-1c6b8ad8b852_isrForCrosstalkSources_1125924_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/ce408f77-801f-42b3-b9eb-1c6b8ad8b852_isrForCrosstalkSources_1125924_42.stderr parsl.dataflow.dflow DEBUG: Task 1673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1673 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1673 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1673 has memoization hash b8cb1ce5ff55fabd4314e70cb6cb82f4 parsl.dataflow.memoization INFO: Task 1673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1673 try 0 launched on executor multi with executor id 1674 parsl.dataflow.dflow INFO: Standard output for task 1673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/8e49d935-6ca0-4c0e-b089-8388841a2d91_isrForCrosstalkSources_1125945_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/8e49d935-6ca0-4c0e-b089-8388841a2d91_isrForCrosstalkSources_1125945_50.stderr parsl.dataflow.dflow DEBUG: Task 1674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1674 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1674 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1674 has memoization hash bd1a51fb52a51d3f3d3dca324e2ad9ca parsl.dataflow.memoization INFO: Task 1674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1674 try 0 launched on executor multi with executor id 1675 parsl.dataflow.dflow INFO: Standard output for task 1674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/870ba7e5-2dce-4ace-bf8c-a917d1d2d794_isrForCrosstalkSources_1125899_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/870ba7e5-2dce-4ace-bf8c-a917d1d2d794_isrForCrosstalkSources_1125899_32.stderr parsl.dataflow.dflow DEBUG: Task 1675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1675 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1675 has memoization hash 9c3f8415f29a9cc58f092647d9126216 parsl.dataflow.memoization INFO: Task 1675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1675 try 0 launched on executor multi with executor id 1676 parsl.dataflow.dflow INFO: Standard output for task 1675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/fda437a3-6967-4c87-aa67-9753e6fe89b6_isrForCrosstalkSources_1125900_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/fda437a3-6967-4c87-aa67-9753e6fe89b6_isrForCrosstalkSources_1125900_11.stderr parsl.dataflow.dflow DEBUG: Task 1676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1676 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1676 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1676 has memoization hash e9abc9a768eb0634ae6bc09858a0df01 parsl.dataflow.memoization INFO: Task 1676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1676 try 0 launched on executor multi with executor id 1677 parsl.dataflow.dflow INFO: Standard output for task 1676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/dba6f628-5793-4a24-907f-9bd84bc5bcdb_isrForCrosstalkSources_1125919_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/dba6f628-5793-4a24-907f-9bd84bc5bcdb_isrForCrosstalkSources_1125919_1.stderr parsl.dataflow.dflow DEBUG: Task 1677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1677 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1677 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1677 has memoization hash 2601d7954fb1669587ce8d9599679e29 parsl.dataflow.memoization INFO: Task 1677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1677 try 0 launched on executor multi with executor id 1678 parsl.dataflow.dflow INFO: Standard output for task 1677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/9a86ce9e-85cd-46d9-8d03-fa834b2df749_isrForCrosstalkSources_1125907_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/9a86ce9e-85cd-46d9-8d03-fa834b2df749_isrForCrosstalkSources_1125907_50.stderr parsl.dataflow.dflow DEBUG: Task 1678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1678 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1678 has memoization hash 7d24f79527de7bca1ed8d7ddeae836d3 parsl.dataflow.memoization INFO: Task 1678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1678 try 0 launched on executor multi with executor id 1679 parsl.dataflow.dflow INFO: Standard output for task 1678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/1f5fc431-705c-4724-a7d4-510a4df43510_isrForCrosstalkSources_1125901_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/1f5fc431-705c-4724-a7d4-510a4df43510_isrForCrosstalkSources_1125901_50.stderr parsl.dataflow.dflow DEBUG: Task 1679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1679 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1679 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1679 has memoization hash 1aacdd74f3dd9cba6a7dd3c305f8cc62 parsl.dataflow.memoization INFO: Task 1679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1679 try 0 launched on executor multi with executor id 1680 parsl.dataflow.dflow INFO: Standard output for task 1679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/db7cd409-0205-4c64-8705-5e5e57112ae5_isrForCrosstalkSources_1125934_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/db7cd409-0205-4c64-8705-5e5e57112ae5_isrForCrosstalkSources_1125934_45.stderr parsl.dataflow.dflow DEBUG: Task 1680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1680 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1680 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1680 has memoization hash 16458e0e5dc5e5bc3cfe676eb8abf860 parsl.dataflow.memoization INFO: Task 1680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1680 try 0 launched on executor multi with executor id 1681 parsl.dataflow.dflow INFO: Standard output for task 1680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/c92c4891-9540-4832-9fbe-beed1f548c7a_isrForCrosstalkSources_1125935_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/c92c4891-9540-4832-9fbe-beed1f548c7a_isrForCrosstalkSources_1125935_32.stderr parsl.dataflow.dflow DEBUG: Task 1681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1681 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1681 has memoization hash 645a62fe97885ce82641f1281194dc9c parsl.dataflow.memoization INFO: Task 1681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1681 try 0 launched on executor multi with executor id 1682 parsl.dataflow.dflow INFO: Standard output for task 1681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/7f0929bf-d86a-4d52-a2e4-6a3302aec38e_isrForCrosstalkSources_1125889_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/7f0929bf-d86a-4d52-a2e4-6a3302aec38e_isrForCrosstalkSources_1125889_43.stderr parsl.dataflow.dflow DEBUG: Task 1682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1682 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1682 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1682 has memoization hash b1a9cfeb01e15a3be70647433c52065e parsl.dataflow.memoization INFO: Task 1682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1682 try 0 launched on executor multi with executor id 1683 parsl.dataflow.dflow INFO: Standard output for task 1682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/336eab5d-4ce1-4c6a-9964-4a29a86baae2_isrForCrosstalkSources_1125921_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/336eab5d-4ce1-4c6a-9964-4a29a86baae2_isrForCrosstalkSources_1125921_42.stderr parsl.dataflow.dflow DEBUG: Task 1683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1683 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1683 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1683 has memoization hash 700af268a76b21b4127b4c8bbe911e4b parsl.dataflow.memoization INFO: Task 1683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1683 try 0 launched on executor multi with executor id 1684 parsl.dataflow.dflow INFO: Standard output for task 1683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/eaf7d378-213e-4b80-8225-5430ad589206_isrForCrosstalkSources_1125913_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/eaf7d378-213e-4b80-8225-5430ad589206_isrForCrosstalkSources_1125913_32.stderr parsl.dataflow.dflow DEBUG: Task 1684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1684 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1684 has memoization hash ff0dbdd733fa373ba2ef491814cdabd0 parsl.dataflow.memoization INFO: Task 1684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1684 try 0 launched on executor multi with executor id 1685 parsl.dataflow.dflow INFO: Standard output for task 1684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/3ea1d29d-44df-4f6b-8fe5-4d802fbb32d6_isrForCrosstalkSources_1125904_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/3ea1d29d-44df-4f6b-8fe5-4d802fbb32d6_isrForCrosstalkSources_1125904_20.stderr parsl.dataflow.dflow DEBUG: Task 1685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1685 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1685 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1685 has memoization hash eb2420ef2c5a7b3eec12c94fce0600b1 parsl.dataflow.memoization INFO: Task 1685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1685 try 0 launched on executor multi with executor id 1686 parsl.dataflow.dflow INFO: Standard output for task 1685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a107e662-a263-408a-b2ad-8001a2ea89c9_isrForCrosstalkSources_1125933_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a107e662-a263-408a-b2ad-8001a2ea89c9_isrForCrosstalkSources_1125933_46.stderr parsl.dataflow.dflow DEBUG: Task 1686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1686 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1686 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1686 has memoization hash a027e92a99ab01524951b7b72e885e09 parsl.dataflow.memoization INFO: Task 1686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1686 try 0 launched on executor multi with executor id 1687 parsl.dataflow.dflow INFO: Standard output for task 1686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/59ccff31-3df7-45fe-93d6-df98c3bede3f_isrForCrosstalkSources_1125909_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/59ccff31-3df7-45fe-93d6-df98c3bede3f_isrForCrosstalkSources_1125909_4.stderr parsl.dataflow.dflow DEBUG: Task 1687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1687 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1687 has memoization hash cf4d08a4bbda9a2ee53e675cb29666f4 parsl.dataflow.memoization INFO: Task 1687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1687 try 0 launched on executor multi with executor id 1688 parsl.dataflow.dflow INFO: Standard output for task 1687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/27b90af0-56b1-4d82-9542-34f87e1ce231_isrForCrosstalkSources_1125923_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/27b90af0-56b1-4d82-9542-34f87e1ce231_isrForCrosstalkSources_1125923_48.stderr parsl.dataflow.dflow DEBUG: Task 1688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1688 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1688 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1688 has memoization hash 09ea8255ff4e0cade9ffbdb839f11b98 parsl.dataflow.memoization INFO: Task 1688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1688 try 0 launched on executor multi with executor id 1689 parsl.dataflow.dflow INFO: Standard output for task 1688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/3c498bb5-3ca9-497a-be25-abf36a89826b_isrForCrosstalkSources_1125901_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/3c498bb5-3ca9-497a-be25-abf36a89826b_isrForCrosstalkSources_1125901_46.stderr parsl.dataflow.dflow DEBUG: Task 1689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1689 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1689 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1689 has memoization hash dcf183011282781e7f62ccbfbb67f919 parsl.dataflow.memoization INFO: Task 1689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1689 try 0 launched on executor multi with executor id 1690 parsl.dataflow.dflow INFO: Standard output for task 1689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/8d81d651-52f4-4e1d-a4d9-612237b45d3f_isrForCrosstalkSources_1125892_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/8d81d651-52f4-4e1d-a4d9-612237b45d3f_isrForCrosstalkSources_1125892_56.stderr parsl.dataflow.dflow DEBUG: Task 1690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1690 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1690 has memoization hash ea91884e29dea69b5a083938d76b857f parsl.dataflow.memoization INFO: Task 1690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1690 try 0 launched on executor multi with executor id 1691 parsl.dataflow.dflow INFO: Standard output for task 1690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/a5c9e032-9c5a-450e-86d9-03a4bebf45b3_isrForCrosstalkSources_1125924_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/a5c9e032-9c5a-450e-86d9-03a4bebf45b3_isrForCrosstalkSources_1125924_62.stderr parsl.dataflow.dflow DEBUG: Task 1691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1691 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1691 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1691 has memoization hash 15c16bbd18f2f36ef2896101b0a5603d parsl.dataflow.memoization INFO: Task 1691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1691 try 0 launched on executor multi with executor id 1692 parsl.dataflow.dflow INFO: Standard output for task 1691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/bd6b667b-7878-46c4-85c6-abdc14a6b803_isrForCrosstalkSources_1125897_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/bd6b667b-7878-46c4-85c6-abdc14a6b803_isrForCrosstalkSources_1125897_19.stderr parsl.dataflow.dflow DEBUG: Task 1692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1692 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1692 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1692 has memoization hash b03d6ebe0575c6200e011d828a0a7834 parsl.dataflow.memoization INFO: Task 1692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1692 try 0 launched on executor multi with executor id 1693 parsl.dataflow.dflow INFO: Standard output for task 1692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/d726fb0b-757b-4358-9854-51779374d35d_isrForCrosstalkSources_1125912_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/d726fb0b-757b-4358-9854-51779374d35d_isrForCrosstalkSources_1125912_19.stderr parsl.dataflow.dflow DEBUG: Task 1693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1693 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1693 has memoization hash 05108ad2651d522d402156d6c2204674 parsl.dataflow.memoization INFO: Task 1693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1693 try 0 launched on executor multi with executor id 1694 parsl.dataflow.dflow INFO: Standard output for task 1693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/645be1fa-4499-4cb0-a11e-d1c05fd908b8_isrForCrosstalkSources_1125901_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/645be1fa-4499-4cb0-a11e-d1c05fd908b8_isrForCrosstalkSources_1125901_19.stderr parsl.dataflow.dflow DEBUG: Task 1694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1694 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1694 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1694 has memoization hash 30ba95904a33e8879b634b1d1e05d252 parsl.dataflow.memoization INFO: Task 1694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1694 try 0 launched on executor multi with executor id 1695 parsl.dataflow.dflow INFO: Standard output for task 1694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/801333d7-00f2-4ffa-b326-1cf22be50e65_isrForCrosstalkSources_1125917_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/801333d7-00f2-4ffa-b326-1cf22be50e65_isrForCrosstalkSources_1125917_15.stderr parsl.dataflow.dflow DEBUG: Task 1695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1695 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1695 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1695 has memoization hash d8741ec4f56322ae8f74345adae336e2 parsl.dataflow.memoization INFO: Task 1695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1695 try 0 launched on executor multi with executor id 1696 parsl.dataflow.dflow INFO: Standard output for task 1695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/fc10dd18-73e1-48e0-94e0-232008dea71b_isrForCrosstalkSources_1125941_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/fc10dd18-73e1-48e0-94e0-232008dea71b_isrForCrosstalkSources_1125941_34.stderr parsl.dataflow.dflow DEBUG: Task 1696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1696 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1696 has memoization hash c8ad9b66e9620108fcd45ff603b4e575 parsl.dataflow.memoization INFO: Task 1696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1696 try 0 launched on executor multi with executor id 1697 parsl.dataflow.dflow INFO: Standard output for task 1696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c44e3bcc-229e-4444-a5e6-ef582b430c0e_isrForCrosstalkSources_1125916_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c44e3bcc-229e-4444-a5e6-ef582b430c0e_isrForCrosstalkSources_1125916_24.stderr parsl.dataflow.dflow DEBUG: Task 1697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1697 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1697 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1697 has memoization hash 5820b6bd2204ffed778c56bafbd34b61 parsl.dataflow.memoization INFO: Task 1697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1697 try 0 launched on executor multi with executor id 1698 parsl.dataflow.dflow INFO: Standard output for task 1697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/80392472-c7a5-43a8-8d78-5de0d1551101_isrForCrosstalkSources_1125913_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/80392472-c7a5-43a8-8d78-5de0d1551101_isrForCrosstalkSources_1125913_20.stderr parsl.dataflow.dflow DEBUG: Task 1698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1698 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1698 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1698 has memoization hash 178550ebb6e941433fd3dc6e0a475f6d parsl.dataflow.memoization INFO: Task 1698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1698 try 0 launched on executor multi with executor id 1699 parsl.dataflow.dflow INFO: Standard output for task 1698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/9affc510-8e67-43fb-b63c-1a27c5a58e62_isrForCrosstalkSources_1125925_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/9affc510-8e67-43fb-b63c-1a27c5a58e62_isrForCrosstalkSources_1125925_22.stderr parsl.dataflow.dflow DEBUG: Task 1699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1699 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1699 has memoization hash ece15dfa40b7e7537a8da1a88c4d781e parsl.dataflow.memoization INFO: Task 1699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1699 try 0 launched on executor multi with executor id 1700 parsl.dataflow.dflow INFO: Standard output for task 1699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/35d2dfc1-007e-4a53-9c81-084a9a081bbc_isrForCrosstalkSources_1125909_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/35d2dfc1-007e-4a53-9c81-084a9a081bbc_isrForCrosstalkSources_1125909_37.stderr parsl.dataflow.dflow DEBUG: Task 1700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1700 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1700 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1700 has memoization hash 88eca61145bc26fd37af58975092913c parsl.dataflow.memoization INFO: Task 1700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1700 try 0 launched on executor multi with executor id 1701 parsl.dataflow.dflow INFO: Standard output for task 1700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/07300104-0dc1-4577-bcfe-91ccaeb482a7_isrForCrosstalkSources_1125939_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/07300104-0dc1-4577-bcfe-91ccaeb482a7_isrForCrosstalkSources_1125939_16.stderr parsl.dataflow.dflow DEBUG: Task 1701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1701 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1701 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1701 has memoization hash 782c805f18fb5e181e7438538472304b parsl.dataflow.memoization INFO: Task 1701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1701 try 0 launched on executor multi with executor id 1702 parsl.dataflow.dflow INFO: Standard output for task 1701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/2bbbb378-3585-4e30-9c81-94bf418c3cdc_isrForCrosstalkSources_1125897_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/2bbbb378-3585-4e30-9c81-94bf418c3cdc_isrForCrosstalkSources_1125897_7.stderr parsl.dataflow.dflow DEBUG: Task 1702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1702 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1702 has memoization hash 9b83b89a0d154e05de310cae9e72917f parsl.dataflow.memoization INFO: Task 1702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1702 try 0 launched on executor multi with executor id 1703 parsl.dataflow.dflow INFO: Standard output for task 1702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/7c22d566-5852-439a-96d6-596bf5b7f17b_isrForCrosstalkSources_1125921_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/7c22d566-5852-439a-96d6-596bf5b7f17b_isrForCrosstalkSources_1125921_10.stderr parsl.dataflow.dflow DEBUG: Task 1703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1703 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1703 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1703 has memoization hash 37c16425d7df34b42009f34cbb8e4b8a parsl.dataflow.memoization INFO: Task 1703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1703 try 0 launched on executor multi with executor id 1704 parsl.dataflow.dflow INFO: Standard output for task 1703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/aa1d4849-95e3-407a-a9c9-e95d63a12a6f_isrForCrosstalkSources_1125940_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/aa1d4849-95e3-407a-a9c9-e95d63a12a6f_isrForCrosstalkSources_1125940_17.stderr parsl.dataflow.dflow DEBUG: Task 1704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1704 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1704 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1704 has memoization hash 4d115b2e45de842a9571bd7455335cc0 parsl.dataflow.memoization INFO: Task 1704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1704 try 0 launched on executor multi with executor id 1705 parsl.dataflow.dflow INFO: Standard output for task 1704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/f20e7325-cf81-45de-9657-d2a8537b9f40_isrForCrosstalkSources_1125928_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/f20e7325-cf81-45de-9657-d2a8537b9f40_isrForCrosstalkSources_1125928_21.stderr parsl.dataflow.dflow DEBUG: Task 1705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1705 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1705 has memoization hash 6447c83483b84b58578a0b2c86216c68 parsl.dataflow.memoization INFO: Task 1705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1705 try 0 launched on executor multi with executor id 1706 parsl.dataflow.dflow INFO: Standard output for task 1705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/cde5c015-8742-473c-af09-9915d99c345c_isrForCrosstalkSources_1125921_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/cde5c015-8742-473c-af09-9915d99c345c_isrForCrosstalkSources_1125921_23.stderr parsl.dataflow.dflow DEBUG: Task 1706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1706 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1706 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1706 has memoization hash de9bf32b0e80cd519d4340a288e59e45 parsl.dataflow.memoization INFO: Task 1706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1706 try 0 launched on executor multi with executor id 1707 parsl.dataflow.dflow INFO: Standard output for task 1706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/b1ad7722-7949-4027-86d3-f7db54ce8d0c_isrForCrosstalkSources_1125922_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/b1ad7722-7949-4027-86d3-f7db54ce8d0c_isrForCrosstalkSources_1125922_27.stderr parsl.dataflow.dflow DEBUG: Task 1707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1707 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1707 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1707 has memoization hash d438fafe2b90391306c1cb543c85682c parsl.dataflow.memoization INFO: Task 1707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1707 try 0 launched on executor multi with executor id 1708 parsl.dataflow.dflow INFO: Standard output for task 1707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/a1892f49-80b3-48e6-bda3-90d0b5be8718_isrForCrosstalkSources_1125912_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/a1892f49-80b3-48e6-bda3-90d0b5be8718_isrForCrosstalkSources_1125912_24.stderr parsl.dataflow.dflow DEBUG: Task 1708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1708 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1708 has memoization hash e3db3e72e39693d578a589cfdee431ff parsl.dataflow.memoization INFO: Task 1708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1708 try 0 launched on executor multi with executor id 1709 parsl.dataflow.dflow INFO: Standard output for task 1708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2e0ee176-9baf-4b01-b889-a5d96737365d_isrForCrosstalkSources_1125933_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2e0ee176-9baf-4b01-b889-a5d96737365d_isrForCrosstalkSources_1125933_36.stderr parsl.dataflow.dflow DEBUG: Task 1709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1709 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1709 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1709 has memoization hash 019737d6b0f8ece9f210fdd9543b87cd parsl.dataflow.memoization INFO: Task 1709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1709 try 0 launched on executor multi with executor id 1710 parsl.dataflow.dflow INFO: Standard output for task 1709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ce8dcdd7-96fe-4652-ac2a-4208463c9bb2_isrForCrosstalkSources_1125939_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ce8dcdd7-96fe-4652-ac2a-4208463c9bb2_isrForCrosstalkSources_1125939_57.stderr parsl.dataflow.dflow DEBUG: Task 1710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1710 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1710 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1710 has memoization hash 25447cd134b0f6365e9d3efc229c4c3c parsl.dataflow.memoization INFO: Task 1710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1710 try 0 launched on executor multi with executor id 1711 parsl.dataflow.dflow INFO: Standard output for task 1710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/b4d13190-f35c-431a-9679-46b1ed1b557d_isrForCrosstalkSources_1125923_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/b4d13190-f35c-431a-9679-46b1ed1b557d_isrForCrosstalkSources_1125923_28.stderr parsl.dataflow.dflow DEBUG: Task 1711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1711 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1711 has memoization hash 9c8ba36c0d9a6040213ddaddc7815bfe parsl.dataflow.memoization INFO: Task 1711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1711 try 0 launched on executor multi with executor id 1712 parsl.dataflow.dflow INFO: Standard output for task 1711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8749d5bb-3c48-4ee3-b3ed-ff8dc872f740_isrForCrosstalkSources_1125944_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8749d5bb-3c48-4ee3-b3ed-ff8dc872f740_isrForCrosstalkSources_1125944_9.stderr parsl.dataflow.dflow DEBUG: Task 1712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1712 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1712 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1712 has memoization hash e4acfa08f9986840b46f2f264bbc957c parsl.dataflow.memoization INFO: Task 1712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1712 try 0 launched on executor multi with executor id 1713 parsl.dataflow.dflow INFO: Standard output for task 1712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/e0b75ad0-267f-4d02-bedc-d10a44b12633_isrForCrosstalkSources_1125924_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/e0b75ad0-267f-4d02-bedc-d10a44b12633_isrForCrosstalkSources_1125924_20.stderr parsl.dataflow.dflow DEBUG: Task 1713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1713 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1713 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1713 has memoization hash 95e0fa4b5b072c027e35e44845e8e355 parsl.dataflow.memoization INFO: Task 1713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1713 try 0 launched on executor multi with executor id 1714 parsl.dataflow.dflow INFO: Standard output for task 1713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/7b12dde5-7387-4bad-b942-647d64e630f7_isrForCrosstalkSources_1125941_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/7b12dde5-7387-4bad-b942-647d64e630f7_isrForCrosstalkSources_1125941_61.stderr parsl.dataflow.dflow DEBUG: Task 1714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1714 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1714 has memoization hash c97eb431be0df7f9e461097dd622b663 parsl.dataflow.memoization INFO: Task 1714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1714 try 0 launched on executor multi with executor id 1715 parsl.dataflow.dflow INFO: Standard output for task 1714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/3a6082fe-0dff-4be8-a0ee-f532c13f8f37_isrForCrosstalkSources_1125889_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/3a6082fe-0dff-4be8-a0ee-f532c13f8f37_isrForCrosstalkSources_1125889_53.stderr parsl.dataflow.dflow DEBUG: Task 1715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1715 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1715 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1715 has memoization hash d7793a05f3cddf96f55cf66e7c387a04 parsl.dataflow.memoization INFO: Task 1715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1715 try 0 launched on executor multi with executor id 1716 parsl.dataflow.dflow INFO: Standard output for task 1715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/94751682-4b57-4990-a466-30a2385561e1_isrForCrosstalkSources_1125919_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/94751682-4b57-4990-a466-30a2385561e1_isrForCrosstalkSources_1125919_50.stderr parsl.dataflow.dflow DEBUG: Task 1716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1716 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1716 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1716 has memoization hash f2eff124752324959b05877ef925faa9 parsl.dataflow.memoization INFO: Task 1716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1716 try 0 launched on executor multi with executor id 1717 parsl.dataflow.dflow INFO: Standard output for task 1716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/ef49f33f-5f58-4e38-863e-b7d96a44dc90_isrForCrosstalkSources_1125898_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/ef49f33f-5f58-4e38-863e-b7d96a44dc90_isrForCrosstalkSources_1125898_30.stderr parsl.dataflow.dflow DEBUG: Task 1717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1717 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1717 has memoization hash 88c44b63e0f0769f2a19612e9acf3a6e parsl.dataflow.memoization INFO: Task 1717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1717 try 0 launched on executor multi with executor id 1718 parsl.dataflow.dflow INFO: Standard output for task 1717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/b949307b-e86e-4edd-a1c2-978b488d9777_isrForCrosstalkSources_1125946_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/b949307b-e86e-4edd-a1c2-978b488d9777_isrForCrosstalkSources_1125946_26.stderr parsl.dataflow.dflow DEBUG: Task 1718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1718 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1718 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1718 has memoization hash 17690b9d3379257d4def4e099f8698ac parsl.dataflow.memoization INFO: Task 1718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1718 try 0 launched on executor multi with executor id 1719 parsl.dataflow.dflow INFO: Standard output for task 1718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/5b25cabe-0e4c-484a-9e16-387af2f72c29_isrForCrosstalkSources_1125933_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/5b25cabe-0e4c-484a-9e16-387af2f72c29_isrForCrosstalkSources_1125933_5.stderr parsl.dataflow.dflow DEBUG: Task 1719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1719 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1719 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1719 has memoization hash 4724c0c302f4db08ce9274c552b36a0c parsl.dataflow.memoization INFO: Task 1719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1719 try 0 launched on executor multi with executor id 1720 parsl.dataflow.dflow INFO: Standard output for task 1719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/da63bccc-99d2-4d2c-8073-3ce56dd164fc_isrForCrosstalkSources_1125920_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/da63bccc-99d2-4d2c-8073-3ce56dd164fc_isrForCrosstalkSources_1125920_19.stderr parsl.dataflow.dflow DEBUG: Task 1720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1720 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1720 has memoization hash a4b3eba3d36f4a3244c11dc57cb57736 parsl.dataflow.memoization INFO: Task 1720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1720 try 0 launched on executor multi with executor id 1721 parsl.dataflow.dflow INFO: Standard output for task 1720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/38c1b795-c023-4c41-a8e9-05fd42e62c7e_isrForCrosstalkSources_1125908_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/38c1b795-c023-4c41-a8e9-05fd42e62c7e_isrForCrosstalkSources_1125908_32.stderr parsl.dataflow.dflow DEBUG: Task 1721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1721 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1721 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1721 has memoization hash 2b2dac01ea2cb9fbcf8999826e5a0770 parsl.dataflow.memoization INFO: Task 1721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1721 try 0 launched on executor multi with executor id 1722 parsl.dataflow.dflow INFO: Standard output for task 1721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/e1af1f4c-8773-4972-95e5-a8bd4c9f62cd_isrForCrosstalkSources_1125926_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/e1af1f4c-8773-4972-95e5-a8bd4c9f62cd_isrForCrosstalkSources_1125926_42.stderr parsl.dataflow.dflow DEBUG: Task 1722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1722 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1722 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1722 has memoization hash 842e5efd1e4fcfbb163c7df07add23bf parsl.dataflow.memoization INFO: Task 1722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1722 try 0 launched on executor multi with executor id 1723 parsl.dataflow.dflow INFO: Standard output for task 1722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/4ee0f4b9-47b8-4831-b167-450fccb4ecfa_isrForCrosstalkSources_1125902_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/4ee0f4b9-47b8-4831-b167-450fccb4ecfa_isrForCrosstalkSources_1125902_35.stderr parsl.dataflow.dflow DEBUG: Task 1723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1723 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1723 has memoization hash 50db344db3f0a892232442fef19853f6 parsl.dataflow.memoization INFO: Task 1723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1723 try 0 launched on executor multi with executor id 1724 parsl.dataflow.dflow INFO: Standard output for task 1723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/af2da7dd-692d-47b7-8cf6-b1787dacd0b3_isrForCrosstalkSources_1125936_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/af2da7dd-692d-47b7-8cf6-b1787dacd0b3_isrForCrosstalkSources_1125936_45.stderr parsl.dataflow.dflow DEBUG: Task 1724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1724 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1724 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1724 has memoization hash 47ec1ebe72f683234a98c1a69b19c76c parsl.dataflow.memoization INFO: Task 1724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1724 try 0 launched on executor multi with executor id 1725 parsl.dataflow.dflow INFO: Standard output for task 1724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/1f5c980e-224c-4b4a-8093-4f516d4ed443_isrForCrosstalkSources_1125911_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/1f5c980e-224c-4b4a-8093-4f516d4ed443_isrForCrosstalkSources_1125911_32.stderr parsl.dataflow.dflow DEBUG: Task 1725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1725 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1725 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1725 has memoization hash 69ccfb38aae81092d18c313d0846a36b parsl.dataflow.memoization INFO: Task 1725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1725 try 0 launched on executor multi with executor id 1726 parsl.dataflow.dflow INFO: Standard output for task 1725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b902671d-0050-40b2-a56d-6520ed0f08dc_isrForCrosstalkSources_1125934_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b902671d-0050-40b2-a56d-6520ed0f08dc_isrForCrosstalkSources_1125934_13.stderr parsl.dataflow.dflow DEBUG: Task 1726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1726 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1726 has memoization hash ab16eab15b67fea77b299bc9197dce3b parsl.dataflow.memoization INFO: Task 1726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1726 try 0 launched on executor multi with executor id 1727 parsl.dataflow.dflow INFO: Standard output for task 1726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/32b0581c-6c92-4f5d-88a5-ad4901cbf3ac_isrForCrosstalkSources_1125904_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/32b0581c-6c92-4f5d-88a5-ad4901cbf3ac_isrForCrosstalkSources_1125904_18.stderr parsl.dataflow.dflow DEBUG: Task 1727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1727 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1727 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1727 has memoization hash 838f089a0896e422ce08dd0d4cdae8c0 parsl.dataflow.memoization INFO: Task 1727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1727 try 0 launched on executor multi with executor id 1728 parsl.dataflow.dflow INFO: Standard output for task 1727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/04195519-6b1a-4ea5-a382-a2d8e193f977_isrForCrosstalkSources_1125898_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/04195519-6b1a-4ea5-a382-a2d8e193f977_isrForCrosstalkSources_1125898_1.stderr parsl.dataflow.dflow DEBUG: Task 1728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1728 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1728 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1728 has memoization hash 6841dcbac5f8089e3334009edd9d92b7 parsl.dataflow.memoization INFO: Task 1728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1728 try 0 launched on executor multi with executor id 1729 parsl.dataflow.dflow INFO: Standard output for task 1728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/f34d3a51-0dfc-48f7-a0b2-2427eaaa06ab_isrForCrosstalkSources_1125936_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/f34d3a51-0dfc-48f7-a0b2-2427eaaa06ab_isrForCrosstalkSources_1125936_3.stderr parsl.dataflow.dflow DEBUG: Task 1729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1729 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1729 has memoization hash 7ec6718e2fc959f379b2ead9c53821f5 parsl.dataflow.memoization INFO: Task 1729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1729 try 0 launched on executor multi with executor id 1730 parsl.dataflow.dflow INFO: Standard output for task 1729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/4a835b6f-7bcd-4437-b10b-89f4cf30ccd7_isrForCrosstalkSources_1125917_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/4a835b6f-7bcd-4437-b10b-89f4cf30ccd7_isrForCrosstalkSources_1125917_19.stderr parsl.dataflow.dflow DEBUG: Task 1730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1730 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1730 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1730 has memoization hash 0e5d2023fda39b37fad8d5153bb0f642 parsl.dataflow.memoization INFO: Task 1730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1730 try 0 launched on executor multi with executor id 1731 parsl.dataflow.dflow INFO: Standard output for task 1730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/8c99813e-85cb-477a-bf1a-a5272998d2ac_isrForCrosstalkSources_1125913_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/8c99813e-85cb-477a-bf1a-a5272998d2ac_isrForCrosstalkSources_1125913_17.stderr parsl.dataflow.dflow DEBUG: Task 1731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1731 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1731 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1731 has memoization hash 2ffee89700fcd483ff3cb767789d6b0f parsl.dataflow.memoization INFO: Task 1731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1731 try 0 launched on executor multi with executor id 1732 parsl.dataflow.dflow INFO: Standard output for task 1731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/71e3685e-26fe-4c52-9f35-06921947cbf8_isrForCrosstalkSources_1125894_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/71e3685e-26fe-4c52-9f35-06921947cbf8_isrForCrosstalkSources_1125894_33.stderr parsl.dataflow.dflow DEBUG: Task 1732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1732 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1732 has memoization hash 3f5abe512caaee57fae6903c260c1ddf parsl.dataflow.memoization INFO: Task 1732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1732 try 0 launched on executor multi with executor id 1733 parsl.dataflow.dflow INFO: Standard output for task 1732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/274774da-d7de-48e0-a5eb-595ccbd3079a_isrForCrosstalkSources_1125916_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/274774da-d7de-48e0-a5eb-595ccbd3079a_isrForCrosstalkSources_1125916_23.stderr parsl.dataflow.dflow DEBUG: Task 1733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1733 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1733 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1733 has memoization hash a846f5370b9c15c3ba4a4903be22a5e8 parsl.dataflow.memoization INFO: Task 1733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1733 try 0 launched on executor multi with executor id 1734 parsl.dataflow.dflow INFO: Standard output for task 1733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/69bcabfa-2d83-4af6-ac4e-ce6de58db6dc_isrForCrosstalkSources_1125926_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/69bcabfa-2d83-4af6-ac4e-ce6de58db6dc_isrForCrosstalkSources_1125926_50.stderr parsl.dataflow.dflow DEBUG: Task 1734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1734 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1734 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1734 has memoization hash 1015324698e5ee71d571b7be9f0e64d6 parsl.dataflow.memoization INFO: Task 1734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1734 try 0 launched on executor multi with executor id 1735 parsl.dataflow.dflow INFO: Standard output for task 1734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/0500b04c-ada3-4eaf-8eb4-9c35f781d930_isrForCrosstalkSources_1125932_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/0500b04c-ada3-4eaf-8eb4-9c35f781d930_isrForCrosstalkSources_1125932_18.stderr parsl.dataflow.dflow DEBUG: Task 1735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1735 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1735 has memoization hash 5482820973ab0fbc49f2359aa0e90f72 parsl.dataflow.memoization INFO: Task 1735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1735 try 0 launched on executor multi with executor id 1736 parsl.dataflow.dflow INFO: Standard output for task 1735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/15e00f63-6670-4b8e-b060-771099601235_isrForCrosstalkSources_1125945_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/15e00f63-6670-4b8e-b060-771099601235_isrForCrosstalkSources_1125945_23.stderr parsl.dataflow.dflow DEBUG: Task 1736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1736 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1736 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1736 has memoization hash badc0d30d913c8e0b5400160ea44c7ec parsl.dataflow.memoization INFO: Task 1736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1736 try 0 launched on executor multi with executor id 1737 parsl.dataflow.dflow INFO: Standard output for task 1736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/126a29ed-c563-4b87-b3b0-cdd27daf9272_isrForCrosstalkSources_1125898_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/126a29ed-c563-4b87-b3b0-cdd27daf9272_isrForCrosstalkSources_1125898_19.stderr parsl.dataflow.dflow DEBUG: Task 1737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1737 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1737 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1737 has memoization hash 92f8913060047c1ec9b7fc16fbe1f409 parsl.dataflow.memoization INFO: Task 1737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1737 try 0 launched on executor multi with executor id 1738 parsl.dataflow.dflow INFO: Standard output for task 1737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/d8839d1e-fa67-41d7-b631-dbb06422ffc3_isrForCrosstalkSources_1125899_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/d8839d1e-fa67-41d7-b631-dbb06422ffc3_isrForCrosstalkSources_1125899_35.stderr parsl.dataflow.dflow DEBUG: Task 1738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1738 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1738 has memoization hash ddccdca0aac6c8e2e9f12710be4ea97d parsl.dataflow.memoization INFO: Task 1738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1738 try 0 launched on executor multi with executor id 1739 parsl.dataflow.dflow INFO: Standard output for task 1738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/fefed67a-a1dd-4614-a7f4-a0c3e41b1dd5_isrForCrosstalkSources_1125932_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/fefed67a-a1dd-4614-a7f4-a0c3e41b1dd5_isrForCrosstalkSources_1125932_55.stderr parsl.dataflow.dflow DEBUG: Task 1739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1739 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1739 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1739 has memoization hash 708b15dfbef75c2a72c63c46667e20ce parsl.dataflow.memoization INFO: Task 1739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a8c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1739 try 0 launched on executor multi with executor id 1740 parsl.dataflow.dflow INFO: Standard output for task 1739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/b6c681bd-80ab-4867-9f64-0d5496516b4f_isrForCrosstalkSources_1125933_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/b6c681bd-80ab-4867-9f64-0d5496516b4f_isrForCrosstalkSources_1125933_57.stderr parsl.dataflow.dflow DEBUG: Task 1740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1740 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1740 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1740 has memoization hash 6ddb6288ba38aa118486bdb280caddb9 parsl.dataflow.memoization INFO: Task 1740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aec0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1740 try 0 launched on executor multi with executor id 1741 parsl.dataflow.dflow INFO: Standard output for task 1740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/dc3faa2a-0c89-4e51-98d6-53ce2a7d5bfa_isrForCrosstalkSources_1125908_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/dc3faa2a-0c89-4e51-98d6-53ce2a7d5bfa_isrForCrosstalkSources_1125908_4.stderr parsl.dataflow.dflow DEBUG: Task 1741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1741 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1741 has memoization hash 6e600f21305fbc96d979eab53a893191 parsl.dataflow.memoization INFO: Task 1741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aec180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1741 try 0 launched on executor multi with executor id 1742 parsl.dataflow.dflow INFO: Standard output for task 1741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/5ee333d5-207c-4f0b-8508-17599068b64a_isrForCrosstalkSources_1125907_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/5ee333d5-207c-4f0b-8508-17599068b64a_isrForCrosstalkSources_1125907_11.stderr parsl.dataflow.dflow DEBUG: Task 1742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1742 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1742 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1742 has memoization hash 3a6776e39b1035b59f7a91ee74275b88 parsl.dataflow.memoization INFO: Task 1742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aec2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1742 try 0 launched on executor multi with executor id 1743 parsl.dataflow.dflow INFO: Standard output for task 1742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/3c0ae549-df8f-4985-8ad5-e33ca7fda282_isrForCrosstalkSources_1125890_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/3c0ae549-df8f-4985-8ad5-e33ca7fda282_isrForCrosstalkSources_1125890_21.stderr parsl.dataflow.dflow DEBUG: Task 1743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1743 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1743 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1743 has memoization hash 9678ebf0fe8bc29fb8924ffd389f4c91 parsl.dataflow.memoization INFO: Task 1743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aec220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1743 try 0 launched on executor multi with executor id 1744 parsl.dataflow.dflow INFO: Standard output for task 1743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/5b5e4708-5bdf-4477-8ab0-a6186ddc6aeb_isrForCrosstalkSources_1125925_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/5b5e4708-5bdf-4477-8ab0-a6186ddc6aeb_isrForCrosstalkSources_1125925_54.stderr parsl.dataflow.dflow DEBUG: Task 1744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1744 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1744 has memoization hash 6de1a5da485e51a2904dc0e77a48cec0 parsl.dataflow.memoization INFO: Task 1744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aec360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1744 try 0 launched on executor multi with executor id 1745 parsl.dataflow.dflow INFO: Standard output for task 1744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/123f3939-22fa-4f2b-a481-77cc6f017560_isrForCrosstalkSources_1125945_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/123f3939-22fa-4f2b-a481-77cc6f017560_isrForCrosstalkSources_1125945_7.stderr parsl.dataflow.dflow DEBUG: Task 1745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1745 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1745 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1745 has memoization hash c9e3b4e5e7c69275032d25b0e2f52da4 parsl.dataflow.memoization INFO: Task 1745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aec400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1745 try 0 launched on executor multi with executor id 1746 parsl.dataflow.dflow INFO: Standard output for task 1745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/971417d0-4f63-4eb2-a2bb-4da6ffeaf907_isrForCrosstalkSources_1125927_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/971417d0-4f63-4eb2-a2bb-4da6ffeaf907_isrForCrosstalkSources_1125927_46.stderr parsl.dataflow.dflow DEBUG: Task 1746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1746 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1746 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1746 has memoization hash 4974ac446c1967707f9dcfd9962735af parsl.dataflow.memoization INFO: Task 1746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aec4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1746 try 0 launched on executor multi with executor id 1747 parsl.dataflow.dflow INFO: Standard output for task 1746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/0575972c-bb80-47b4-8ddd-c96ebcb0511f_isrForCrosstalkSources_1125926_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/0575972c-bb80-47b4-8ddd-c96ebcb0511f_isrForCrosstalkSources_1125926_54.stderr parsl.dataflow.dflow DEBUG: Task 1747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1747 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1747 has memoization hash 36b9b950608f1f165788bdadd25da155 parsl.dataflow.memoization INFO: Task 1747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aec5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1747 try 0 launched on executor multi with executor id 1748 parsl.dataflow.dflow INFO: Standard output for task 1747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/587a884d-f757-4662-ab2a-744174c93b9e_isrForCrosstalkSources_1125912_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/587a884d-f757-4662-ab2a-744174c93b9e_isrForCrosstalkSources_1125912_54.stderr parsl.dataflow.dflow DEBUG: Task 1748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1748 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1748 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1748 has memoization hash 57cd7428125de8b99e0aeaa9cb6d8c6e parsl.dataflow.memoization INFO: Task 1748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aec540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1748 try 0 launched on executor multi with executor id 1749 parsl.dataflow.dflow INFO: Standard output for task 1748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/dc3fa744-afcb-48a0-a711-7edf537a098f_isrForCrosstalkSources_1125893_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/dc3fa744-afcb-48a0-a711-7edf537a098f_isrForCrosstalkSources_1125893_44.stderr parsl.dataflow.dflow DEBUG: Task 1749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1749 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1749 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1749 has memoization hash c2dd25750280f8ba1838f25822e0a7bc parsl.dataflow.memoization INFO: Task 1749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aec680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1749 try 0 launched on executor multi with executor id 1750 parsl.dataflow.dflow INFO: Standard output for task 1749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/4635d720-703c-4df4-91f7-fbb697e89f16_isrForCrosstalkSources_1125918_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/4635d720-703c-4df4-91f7-fbb697e89f16_isrForCrosstalkSources_1125918_36.stderr parsl.dataflow.dflow DEBUG: Task 1750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1750 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1750 has memoization hash 3665eda571ccb8020cb6b63e3a078641 parsl.dataflow.memoization INFO: Task 1750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aec720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1750 try 0 launched on executor multi with executor id 1751 parsl.dataflow.dflow INFO: Standard output for task 1750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c52f7727-54eb-4639-85c5-8aeaa57f9ccf_isrForCrosstalkSources_1125916_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c52f7727-54eb-4639-85c5-8aeaa57f9ccf_isrForCrosstalkSources_1125916_43.stderr parsl.dataflow.dflow DEBUG: Task 1751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1751 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1751 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1751 has memoization hash 23da9fe904bb68d4fddea2da35f3a3b2 parsl.dataflow.memoization INFO: Task 1751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aec7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1751 try 0 launched on executor multi with executor id 1752 parsl.dataflow.dflow INFO: Standard output for task 1751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/35d9ac3c-237d-4d54-b291-9249ad11241c_isrForCrosstalkSources_1125927_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/35d9ac3c-237d-4d54-b291-9249ad11241c_isrForCrosstalkSources_1125927_17.stderr parsl.dataflow.dflow DEBUG: Task 1752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1752 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1752 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1752 has memoization hash 5764ad34f8617ad7e3c46c9dfead0fba parsl.dataflow.memoization INFO: Task 1752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aec900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1752 try 0 launched on executor multi with executor id 1753 parsl.dataflow.dflow INFO: Standard output for task 1752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/baeb05e4-5b05-4364-b1fb-62aad4ac337d_isrForCrosstalkSources_1125917_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/baeb05e4-5b05-4364-b1fb-62aad4ac337d_isrForCrosstalkSources_1125917_9.stderr parsl.dataflow.dflow DEBUG: Task 1753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1753 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1753 has memoization hash da8f242a5d12262b3495f40b3028ef50 parsl.dataflow.memoization INFO: Task 1753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aec860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1753 try 0 launched on executor multi with executor id 1754 parsl.dataflow.dflow INFO: Standard output for task 1753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/1c4aa3ed-7800-48da-b2cb-9264048a67f6_isrForCrosstalkSources_1125923_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/1c4aa3ed-7800-48da-b2cb-9264048a67f6_isrForCrosstalkSources_1125923_57.stderr parsl.dataflow.dflow DEBUG: Task 1754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1754 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1754 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1754 has memoization hash 794bd96ecd6ae9fa9face1fca1fcf9ae parsl.dataflow.memoization INFO: Task 1754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aec9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1754 try 0 launched on executor multi with executor id 1755 parsl.dataflow.dflow INFO: Standard output for task 1754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/bd63b5fd-6fb3-4b69-987d-987cc1b55358_isrForCrosstalkSources_1125916_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/bd63b5fd-6fb3-4b69-987d-987cc1b55358_isrForCrosstalkSources_1125916_14.stderr parsl.dataflow.dflow DEBUG: Task 1755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1755 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1755 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1755 has memoization hash 1146c76253cd0d0ff356a97dd242f6eb parsl.dataflow.memoization INFO: Task 1755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aeca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1755 try 0 launched on executor multi with executor id 1756 parsl.dataflow.dflow INFO: Standard output for task 1755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/c82e60dd-83a2-4c26-ba4e-54c954884d6e_isrForCrosstalkSources_1125927_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/c82e60dd-83a2-4c26-ba4e-54c954884d6e_isrForCrosstalkSources_1125927_51.stderr parsl.dataflow.dflow DEBUG: Task 1756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1756 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1756 has memoization hash 87af097fe4c6d5efb970ac6ab076c421 parsl.dataflow.memoization INFO: Task 1756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aecae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1756 try 0 launched on executor multi with executor id 1757 parsl.dataflow.dflow INFO: Standard output for task 1756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/a0dbb88b-772d-4749-9aaa-29774a7c1461_isrForCrosstalkSources_1125890_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/a0dbb88b-772d-4749-9aaa-29774a7c1461_isrForCrosstalkSources_1125890_60.stderr parsl.dataflow.dflow DEBUG: Task 1757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1757 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1757 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1757 has memoization hash 656ec23df3c0fa03dae07783f8b2f2a7 parsl.dataflow.memoization INFO: Task 1757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aecc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1757 try 0 launched on executor multi with executor id 1758 parsl.dataflow.dflow INFO: Standard output for task 1757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/cd5e0332-7212-40a9-94a0-9a96fc95c958_isrForCrosstalkSources_1125923_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/cd5e0332-7212-40a9-94a0-9a96fc95c958_isrForCrosstalkSources_1125923_41.stderr parsl.dataflow.dflow DEBUG: Task 1758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1758 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1758 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1758 has memoization hash ab92d7f74571ae6f373bbe1686568ae6 parsl.dataflow.memoization INFO: Task 1758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aecb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1758 try 0 launched on executor multi with executor id 1759 parsl.dataflow.dflow INFO: Standard output for task 1758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/99e9f611-b8a4-464b-bcf5-8b5c5854fc55_isrForCrosstalkSources_1125929_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/99e9f611-b8a4-464b-bcf5-8b5c5854fc55_isrForCrosstalkSources_1125929_44.stderr parsl.dataflow.dflow DEBUG: Task 1759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1759 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1759 has memoization hash b86594dd2819970021ce65f7bf8b48c1 parsl.dataflow.memoization INFO: Task 1759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aeccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1759 try 0 launched on executor multi with executor id 1760 parsl.dataflow.dflow INFO: Standard output for task 1759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/15965065-b664-4ffd-867b-898aa72d4ee8_isrForCrosstalkSources_1125892_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/15965065-b664-4ffd-867b-898aa72d4ee8_isrForCrosstalkSources_1125892_22.stderr parsl.dataflow.dflow DEBUG: Task 1760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1760 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1760 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1760 has memoization hash 6a0dbda7465dab26484f53b43ef0b43f parsl.dataflow.memoization INFO: Task 1760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aecd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1760 try 0 launched on executor multi with executor id 1761 parsl.dataflow.dflow INFO: Standard output for task 1760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/9ed35b3d-d197-4ac7-87dc-0f4b2fe6b876_isrForCrosstalkSources_1125895_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/9ed35b3d-d197-4ac7-87dc-0f4b2fe6b876_isrForCrosstalkSources_1125895_19.stderr parsl.dataflow.dflow DEBUG: Task 1761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1761 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1761 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1761 has memoization hash 98b0a39a4f955322fb1476702a3ec6a4 parsl.dataflow.memoization INFO: Task 1761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aece00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1761 try 0 launched on executor multi with executor id 1762 parsl.dataflow.dflow INFO: Standard output for task 1761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/6cf239e9-c6bd-40a3-a8ae-f825134229a7_isrForCrosstalkSources_1125929_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/6cf239e9-c6bd-40a3-a8ae-f825134229a7_isrForCrosstalkSources_1125929_39.stderr parsl.dataflow.dflow DEBUG: Task 1762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1762 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1762 has memoization hash 0119603ed6834503033fa00d12b9012b parsl.dataflow.memoization INFO: Task 1762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aecf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1762 try 0 launched on executor multi with executor id 1763 parsl.dataflow.dflow INFO: Standard output for task 1762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/dcc2b37b-de12-4ee8-9270-99bdaaa69769_isrForCrosstalkSources_1125898_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/dcc2b37b-de12-4ee8-9270-99bdaaa69769_isrForCrosstalkSources_1125898_60.stderr parsl.dataflow.dflow DEBUG: Task 1763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1763 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1763 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1763 has memoization hash 049c05187e4380c14174e02c70c64ec7 parsl.dataflow.memoization INFO: Task 1763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aecea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1763 try 0 launched on executor multi with executor id 1764 parsl.dataflow.dflow INFO: Standard output for task 1763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/666c7029-d787-4958-abd1-7aaf99b7c409_isrForCrosstalkSources_1125939_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/666c7029-d787-4958-abd1-7aaf99b7c409_isrForCrosstalkSources_1125939_59.stderr parsl.dataflow.dflow DEBUG: Task 1764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1764 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1764 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1764 has memoization hash b72ea68568b516878d2dec1923304e97 parsl.dataflow.memoization INFO: Task 1764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aecfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1764 try 0 launched on executor multi with executor id 1765 parsl.dataflow.dflow INFO: Standard output for task 1764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/89baa68d-117d-4f69-aa6d-ba25095a7379_isrForCrosstalkSources_1125929_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/89baa68d-117d-4f69-aa6d-ba25095a7379_isrForCrosstalkSources_1125929_33.stderr parsl.dataflow.dflow DEBUG: Task 1765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1765 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1765 has memoization hash 418a079650114100634eeac93bfbcebb parsl.dataflow.memoization INFO: Task 1765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aed080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1765 try 0 launched on executor multi with executor id 1766 parsl.dataflow.dflow INFO: Standard output for task 1765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/1e0af3b9-3508-4945-a6be-d6861f46a4b3_isrForCrosstalkSources_1125940_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/1e0af3b9-3508-4945-a6be-d6861f46a4b3_isrForCrosstalkSources_1125940_9.stderr parsl.dataflow.dflow DEBUG: Task 1766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1766 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1766 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1766 has memoization hash 331707334b8fe35d0f2bb9b08d6cfcf2 parsl.dataflow.memoization INFO: Task 1766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aed120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1766 try 0 launched on executor multi with executor id 1767 parsl.dataflow.dflow INFO: Standard output for task 1766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/0fdba73d-44a8-4ce6-add9-520bcc2c27b0_isrForCrosstalkSources_1125931_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/0fdba73d-44a8-4ce6-add9-520bcc2c27b0_isrForCrosstalkSources_1125931_13.stderr parsl.dataflow.dflow DEBUG: Task 1767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1767 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1767 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1767 has memoization hash 216ab1df4a221f41c49f222d11c39dc9 parsl.dataflow.memoization INFO: Task 1767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aed260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1767 try 0 launched on executor multi with executor id 1768 parsl.dataflow.dflow INFO: Standard output for task 1767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/35cc4aa9-86e2-4788-8f95-ece4cb20329a_isrForCrosstalkSources_1125916_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/35cc4aa9-86e2-4788-8f95-ece4cb20329a_isrForCrosstalkSources_1125916_29.stderr parsl.dataflow.dflow DEBUG: Task 1768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1768 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1768 has memoization hash 7b14a281c10829d180293a964f0c03ad parsl.dataflow.memoization INFO: Task 1768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aed1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1768 try 0 launched on executor multi with executor id 1769 parsl.dataflow.dflow INFO: Standard output for task 1768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0f5f0ee4-2cbe-4f5d-9f72-1065fef933fb_isrForCrosstalkSources_1125890_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0f5f0ee4-2cbe-4f5d-9f72-1065fef933fb_isrForCrosstalkSources_1125890_27.stderr parsl.dataflow.dflow DEBUG: Task 1769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1769 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1769 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1769 has memoization hash e3a6b7a154ed09613014f9e1cffdb2b1 parsl.dataflow.memoization INFO: Task 1769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aed300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1769 try 0 launched on executor multi with executor id 1770 parsl.dataflow.dflow INFO: Standard output for task 1769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/671682bf-a5ac-4367-813f-fb10a8c11393_isrForCrosstalkSources_1125934_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/671682bf-a5ac-4367-813f-fb10a8c11393_isrForCrosstalkSources_1125934_24.stderr parsl.dataflow.dflow DEBUG: Task 1770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1770 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1770 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1770 has memoization hash 03086eaa0992a822ee33ae9692183139 parsl.dataflow.memoization INFO: Task 1770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aed3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1770 try 0 launched on executor multi with executor id 1771 parsl.dataflow.dflow INFO: Standard output for task 1770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/4131f3ee-5563-4808-92f4-e2483b3e4e9a_isrForCrosstalkSources_1125897_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/4131f3ee-5563-4808-92f4-e2483b3e4e9a_isrForCrosstalkSources_1125897_58.stderr parsl.dataflow.dflow DEBUG: Task 1771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1771 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1771 has memoization hash c4fab33782be59ad758a86642c194433 parsl.dataflow.memoization INFO: Task 1771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aed440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1771 try 0 launched on executor multi with executor id 1772 parsl.dataflow.dflow INFO: Standard output for task 1771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/8527924a-53eb-47bf-a3f2-8fa41510b0d1_isrForCrosstalkSources_1125911_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/8527924a-53eb-47bf-a3f2-8fa41510b0d1_isrForCrosstalkSources_1125911_42.stderr parsl.dataflow.dflow DEBUG: Task 1772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1772 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1772 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1772 has memoization hash 642293ccec56d277667e81f19ca46129 parsl.dataflow.memoization INFO: Task 1772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aed580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1772 try 0 launched on executor multi with executor id 1773 parsl.dataflow.dflow INFO: Standard output for task 1772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/cbfda164-f06f-4215-8220-9e66e949fe11_isrForCrosstalkSources_1125927_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/cbfda164-f06f-4215-8220-9e66e949fe11_isrForCrosstalkSources_1125927_37.stderr parsl.dataflow.dflow DEBUG: Task 1773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1773 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1773 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1773 has memoization hash 06909f1f9c2a7bb3ce4866a2dcf6f3d8 parsl.dataflow.memoization INFO: Task 1773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aed4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1773 try 0 launched on executor multi with executor id 1774 parsl.dataflow.dflow INFO: Standard output for task 1773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/7e5433e0-8aa2-429d-a6a8-1bd0993f39e5_isrForCrosstalkSources_1125896_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/7e5433e0-8aa2-429d-a6a8-1bd0993f39e5_isrForCrosstalkSources_1125896_61.stderr parsl.dataflow.dflow DEBUG: Task 1774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1774 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1774 has memoization hash 48ddca6fa233828b0d187a17c59e9f14 parsl.dataflow.memoization INFO: Task 1774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aed620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1774 try 0 launched on executor multi with executor id 1775 parsl.dataflow.dflow INFO: Standard output for task 1774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f518115f-87c9-4bbf-a6c7-e42c309490b9_isrForCrosstalkSources_1125930_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f518115f-87c9-4bbf-a6c7-e42c309490b9_isrForCrosstalkSources_1125930_59.stderr parsl.dataflow.dflow DEBUG: Task 1775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1775 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1775 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1775 has memoization hash 6d25c507e0c72a011caf7a9a88eda060 parsl.dataflow.memoization INFO: Task 1775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aed6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1775 try 0 launched on executor multi with executor id 1776 parsl.dataflow.dflow INFO: Standard output for task 1775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/ecc09370-a413-439e-9355-3a96d884b1c5_isrForCrosstalkSources_1125903_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/ecc09370-a413-439e-9355-3a96d884b1c5_isrForCrosstalkSources_1125903_37.stderr parsl.dataflow.dflow DEBUG: Task 1776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1776 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1776 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1776 has memoization hash d0956f8b33e16d9c205ad161e8e3c2b9 parsl.dataflow.memoization INFO: Task 1776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aed760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1776 try 0 launched on executor multi with executor id 1777 parsl.dataflow.dflow INFO: Standard output for task 1776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/bc3a1b35-93dc-413f-9004-f8874b61e4c8_isrForCrosstalkSources_1125943_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/bc3a1b35-93dc-413f-9004-f8874b61e4c8_isrForCrosstalkSources_1125943_49.stderr parsl.dataflow.dflow DEBUG: Task 1777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1777 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1777 has memoization hash fbde2ccba516fd4eb62f81217dc639f5 parsl.dataflow.memoization INFO: Task 1777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aed8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1777 try 0 launched on executor multi with executor id 1778 parsl.dataflow.dflow INFO: Standard output for task 1777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/41ce4bee-b835-4171-b2a6-ba68b98c835a_isrForCrosstalkSources_1125897_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/41ce4bee-b835-4171-b2a6-ba68b98c835a_isrForCrosstalkSources_1125897_55.stderr parsl.dataflow.dflow DEBUG: Task 1778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1778 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1778 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1778 has memoization hash 5f42dd068e1acf76178a4dc3805d3c81 parsl.dataflow.memoization INFO: Task 1778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aed800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1778 try 0 launched on executor multi with executor id 1779 parsl.dataflow.dflow INFO: Standard output for task 1778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/08ba6059-d5b7-4910-af73-04f669ea2673_isrForCrosstalkSources_1125913_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/08ba6059-d5b7-4910-af73-04f669ea2673_isrForCrosstalkSources_1125913_4.stderr parsl.dataflow.dflow DEBUG: Task 1779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1779 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1779 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1779 has memoization hash 5e24ff427bbf4976945fc7a787f768a5 parsl.dataflow.memoization INFO: Task 1779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aed940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1779 try 0 launched on executor multi with executor id 1780 parsl.dataflow.dflow INFO: Standard output for task 1779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/bd9fc695-1ce2-4c78-8778-3abaccb22b07_isrForCrosstalkSources_1125936_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/bd9fc695-1ce2-4c78-8778-3abaccb22b07_isrForCrosstalkSources_1125936_56.stderr parsl.dataflow.dflow DEBUG: Task 1780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1780 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1780 has memoization hash decb2c79ef3d78a134ec9a2039b24638 parsl.dataflow.memoization INFO: Task 1780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aed9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1780 try 0 launched on executor multi with executor id 1781 parsl.dataflow.dflow INFO: Standard output for task 1780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/0bc50006-6e75-4763-9c41-af0912d2b3db_isrForCrosstalkSources_1125913_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/0bc50006-6e75-4763-9c41-af0912d2b3db_isrForCrosstalkSources_1125913_25.stderr parsl.dataflow.dflow DEBUG: Task 1781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1781 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1781 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1781 has memoization hash a1875f8b982c0e596b6ac268eb00a16a parsl.dataflow.memoization INFO: Task 1781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aeda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1781 try 0 launched on executor multi with executor id 1782 parsl.dataflow.dflow INFO: Standard output for task 1781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/fd2f334f-114e-496b-8f1d-cae93b1f6362_isrForCrosstalkSources_1125911_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/fd2f334f-114e-496b-8f1d-cae93b1f6362_isrForCrosstalkSources_1125911_56.stderr parsl.dataflow.dflow DEBUG: Task 1782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1782 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1782 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1782 has memoization hash 3243701e8cf79859aa691c744a9eeba4 parsl.dataflow.memoization INFO: Task 1782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aedbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1782 try 0 launched on executor multi with executor id 1783 parsl.dataflow.dflow INFO: Standard output for task 1782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/04a89edb-f49c-4077-94fb-4e701b984b40_isrForCrosstalkSources_1125940_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/04a89edb-f49c-4077-94fb-4e701b984b40_isrForCrosstalkSources_1125940_61.stderr parsl.dataflow.dflow DEBUG: Task 1783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1783 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1783 has memoization hash 47abc9729bea2beeb691ffed3bbf9666 parsl.dataflow.memoization INFO: Task 1783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aedb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1783 try 0 launched on executor multi with executor id 1784 parsl.dataflow.dflow INFO: Standard output for task 1783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/27aa430e-4282-49a3-8024-b056b6ae21d6_isrForCrosstalkSources_1125947_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/27aa430e-4282-49a3-8024-b056b6ae21d6_isrForCrosstalkSources_1125947_21.stderr parsl.dataflow.dflow DEBUG: Task 1784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1784 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1784 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1784 has memoization hash 99088b939bf8462d3a25524c1300cd66 parsl.dataflow.memoization INFO: Task 1784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aedc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1784 try 0 launched on executor multi with executor id 1785 parsl.dataflow.dflow INFO: Standard output for task 1784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/dc5ab3c5-0931-4614-9189-6612e616be24_isrForCrosstalkSources_1125947_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/dc5ab3c5-0931-4614-9189-6612e616be24_isrForCrosstalkSources_1125947_42.stderr parsl.dataflow.dflow DEBUG: Task 1785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1785 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1785 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1785 has memoization hash 04be10ab600837650af3ffb38cd5b5c8 parsl.dataflow.memoization INFO: Task 1785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aedd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1785 try 0 launched on executor multi with executor id 1786 parsl.dataflow.dflow INFO: Standard output for task 1785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/18ff9599-06fa-4828-baff-80ad62b1adb8_isrForCrosstalkSources_1125911_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/18ff9599-06fa-4828-baff-80ad62b1adb8_isrForCrosstalkSources_1125911_21.stderr parsl.dataflow.dflow DEBUG: Task 1786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1786 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1786 has memoization hash 03e57df19e06367bf5628638d54aabeb parsl.dataflow.memoization INFO: Task 1786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aedda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1786 try 0 launched on executor multi with executor id 1787 parsl.dataflow.dflow INFO: Standard output for task 1786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/eadb659d-b568-47f6-8f44-7f7ff8f88f15_isrForCrosstalkSources_1125913_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/eadb659d-b568-47f6-8f44-7f7ff8f88f15_isrForCrosstalkSources_1125913_43.stderr parsl.dataflow.dflow DEBUG: Task 1787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1787 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1787 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1787 has memoization hash 49c04e308e69b62fe9eff69b38d2068d parsl.dataflow.memoization INFO: Task 1787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aedee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1787 try 0 launched on executor multi with executor id 1788 parsl.dataflow.dflow INFO: Standard output for task 1787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c421210f-58bf-444b-b5df-3f969d34cc79_isrForCrosstalkSources_1125904_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c421210f-58bf-444b-b5df-3f969d34cc79_isrForCrosstalkSources_1125904_45.stderr parsl.dataflow.dflow DEBUG: Task 1788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1788 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1788 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1788 has memoization hash c0f5b8cc8d0cedc85ea0741f4828d84b parsl.dataflow.memoization INFO: Task 1788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aede40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1788 try 0 launched on executor multi with executor id 1789 parsl.dataflow.dflow INFO: Standard output for task 1788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/af022f8e-a17c-4134-a62d-fca668882539_isrForCrosstalkSources_1125911_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/af022f8e-a17c-4134-a62d-fca668882539_isrForCrosstalkSources_1125911_8.stderr parsl.dataflow.dflow DEBUG: Task 1789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1789 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1789 has memoization hash f1546536e2d441fc530ca712747b6666 parsl.dataflow.memoization INFO: Task 1789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aedf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1789 try 0 launched on executor multi with executor id 1790 parsl.dataflow.dflow INFO: Standard output for task 1789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/8b380154-b8e2-4d3b-8245-04c45775450b_isrForCrosstalkSources_1125928_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/8b380154-b8e2-4d3b-8245-04c45775450b_isrForCrosstalkSources_1125928_34.stderr parsl.dataflow.dflow DEBUG: Task 1790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1790 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1790 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1790 has memoization hash cbec8d06c5d087239b8806c8ca4c9a62 parsl.dataflow.memoization INFO: Task 1790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aee020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1790 try 0 launched on executor multi with executor id 1791 parsl.dataflow.dflow INFO: Standard output for task 1790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2d7720a8-7059-45ca-a86e-fc95e03ba0ab_isrForCrosstalkSources_1125895_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2d7720a8-7059-45ca-a86e-fc95e03ba0ab_isrForCrosstalkSources_1125895_2.stderr parsl.dataflow.dflow DEBUG: Task 1791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1791 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1791 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1791 has memoization hash f736999b820faf5f7a20e3fb9298f8e7 parsl.dataflow.memoization INFO: Task 1791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aee0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1791 try 0 launched on executor multi with executor id 1792 parsl.dataflow.dflow INFO: Standard output for task 1791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/a9bb2708-6789-4ff1-a9a1-aa33f74f6f55_isrForCrosstalkSources_1125934_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/a9bb2708-6789-4ff1-a9a1-aa33f74f6f55_isrForCrosstalkSources_1125934_49.stderr parsl.dataflow.dflow DEBUG: Task 1792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1792 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1792 has memoization hash c71ebcd9b3ae12e06a46aa0dc8d6ba2e parsl.dataflow.memoization INFO: Task 1792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aee200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1792 try 0 launched on executor multi with executor id 1793 parsl.dataflow.dflow INFO: Standard output for task 1792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/0827ffb1-eed8-42f7-ad5e-bd70053d574c_isrForCrosstalkSources_1125924_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/0827ffb1-eed8-42f7-ad5e-bd70053d574c_isrForCrosstalkSources_1125924_36.stderr parsl.dataflow.dflow DEBUG: Task 1793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1793 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1793 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1793 has memoization hash ce2780a9e8bf8bdb2242673b204a8e2d parsl.dataflow.memoization INFO: Task 1793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aee160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1793 try 0 launched on executor multi with executor id 1794 parsl.dataflow.dflow INFO: Standard output for task 1793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/baaeeb97-fe53-4613-a71c-c0a40fa6fe46_isrForCrosstalkSources_1125934_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/baaeeb97-fe53-4613-a71c-c0a40fa6fe46_isrForCrosstalkSources_1125934_47.stderr parsl.dataflow.dflow DEBUG: Task 1794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1794 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1794 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1794 has memoization hash 74e1eabf74312ae98369a99f0ff136fe parsl.dataflow.memoization INFO: Task 1794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aee2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1794 try 0 launched on executor multi with executor id 1795 parsl.dataflow.dflow INFO: Standard output for task 1794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/4f75c51b-7e72-4df2-b996-f99662209f69_isrForCrosstalkSources_1125930_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/4f75c51b-7e72-4df2-b996-f99662209f69_isrForCrosstalkSources_1125930_39.stderr parsl.dataflow.dflow DEBUG: Task 1795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1795 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1795 has memoization hash f503fe8a7367c818a8d32ca089cfc222 parsl.dataflow.memoization INFO: Task 1795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aee340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1795 try 0 launched on executor multi with executor id 1796 parsl.dataflow.dflow INFO: Standard output for task 1795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/5d8d597c-2930-44d7-93bf-1c49531d683f_isrForCrosstalkSources_1125900_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/5d8d597c-2930-44d7-93bf-1c49531d683f_isrForCrosstalkSources_1125900_51.stderr parsl.dataflow.dflow DEBUG: Task 1796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1796 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1796 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1796 has memoization hash ad2e9283251760f2bbed96f3847d9491 parsl.dataflow.memoization INFO: Task 1796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aee3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1796 try 0 launched on executor multi with executor id 1797 parsl.dataflow.dflow INFO: Standard output for task 1796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/f953671d-b506-430c-83de-1dba4f729674_isrForCrosstalkSources_1125899_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/f953671d-b506-430c-83de-1dba4f729674_isrForCrosstalkSources_1125899_9.stderr parsl.dataflow.dflow DEBUG: Task 1797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1797 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1797 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1797 has memoization hash dbf42c025bfe19a0041e594026aa0576 parsl.dataflow.memoization INFO: Task 1797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aee520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1797 try 0 launched on executor multi with executor id 1798 parsl.dataflow.dflow INFO: Standard output for task 1797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/ee8f2feb-e058-45cf-b6d8-16140b6fe05d_isrForCrosstalkSources_1125945_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/ee8f2feb-e058-45cf-b6d8-16140b6fe05d_isrForCrosstalkSources_1125945_47.stderr parsl.dataflow.dflow DEBUG: Task 1798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1798 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1798 has memoization hash 3236b40a5b1919d9cbbbbdb6d3015491 parsl.dataflow.memoization INFO: Task 1798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aee480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1798 try 0 launched on executor multi with executor id 1799 parsl.dataflow.dflow INFO: Standard output for task 1798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a4f0d649-be2d-42fc-8d01-da9ea118b70b_isrForCrosstalkSources_1125909_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a4f0d649-be2d-42fc-8d01-da9ea118b70b_isrForCrosstalkSources_1125909_9.stderr parsl.dataflow.dflow DEBUG: Task 1799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1799 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1799 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1799 has memoization hash af4636e7348de0738c76235692aab650 parsl.dataflow.memoization INFO: Task 1799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aee5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1799 try 0 launched on executor multi with executor id 1800 parsl.dataflow.dflow INFO: Standard output for task 1799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/4080a70f-74d4-4b1d-a16f-c092d6ac561c_isrForCrosstalkSources_1125891_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/4080a70f-74d4-4b1d-a16f-c092d6ac561c_isrForCrosstalkSources_1125891_56.stderr parsl.dataflow.dflow DEBUG: Task 1800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1800 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1800 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1800 has memoization hash a11c0be247517e8e05df52c0a9df02b0 parsl.dataflow.memoization INFO: Task 1800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aee660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1800 try 0 launched on executor multi with executor id 1801 parsl.dataflow.dflow INFO: Standard output for task 1800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/2eee72d2-2b99-46a2-bf8e-e86fd32099f8_isrForCrosstalkSources_1125935_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/2eee72d2-2b99-46a2-bf8e-e86fd32099f8_isrForCrosstalkSources_1125935_1.stderr parsl.dataflow.dflow DEBUG: Task 1801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1801 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1801 has memoization hash bbd81d0bf8b14c72e91539551ab01071 parsl.dataflow.memoization INFO: Task 1801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aee700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1801 try 0 launched on executor multi with executor id 1802 parsl.dataflow.dflow INFO: Standard output for task 1801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/9947b9d8-572a-4e53-9c61-37b2ffc53c58_isrForCrosstalkSources_1125943_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/9947b9d8-572a-4e53-9c61-37b2ffc53c58_isrForCrosstalkSources_1125943_16.stderr parsl.dataflow.dflow DEBUG: Task 1802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1802 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1802 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1802 has memoization hash 1e1e11f32fdef7b1320f71e4e785038c parsl.dataflow.memoization INFO: Task 1802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aee840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1802 try 0 launched on executor multi with executor id 1803 parsl.dataflow.dflow INFO: Standard output for task 1802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/bf3d4b58-8851-42a5-bb98-4b7d65da99be_isrForCrosstalkSources_1125903_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/bf3d4b58-8851-42a5-bb98-4b7d65da99be_isrForCrosstalkSources_1125903_50.stderr parsl.dataflow.dflow DEBUG: Task 1803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1803 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1803 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1803 has memoization hash 5ba6b1777d6438f05621e0f2eacc1fc2 parsl.dataflow.memoization INFO: Task 1803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aee7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1803 try 0 launched on executor multi with executor id 1804 parsl.dataflow.dflow INFO: Standard output for task 1803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/76ba96a1-7271-4d0d-853e-5dac62824692_isrForCrosstalkSources_1125913_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/76ba96a1-7271-4d0d-853e-5dac62824692_isrForCrosstalkSources_1125913_36.stderr parsl.dataflow.dflow DEBUG: Task 1804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1804 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1804 has memoization hash a2fe3b7bba413a022a0bbc75c561d702 parsl.dataflow.memoization INFO: Task 1804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aee8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1804 try 0 launched on executor multi with executor id 1805 parsl.dataflow.dflow INFO: Standard output for task 1804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/cc2f2bcc-9c35-447e-a663-53b043a72af2_isrForCrosstalkSources_1125927_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/cc2f2bcc-9c35-447e-a663-53b043a72af2_isrForCrosstalkSources_1125927_6.stderr parsl.dataflow.dflow DEBUG: Task 1805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1805 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1805 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1805 has memoization hash 73f79d78d86fda8c0532f8730696a6ff parsl.dataflow.memoization INFO: Task 1805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aee980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1805 try 0 launched on executor multi with executor id 1806 parsl.dataflow.dflow INFO: Standard output for task 1805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/d486716f-58cf-4a9a-85af-7a3e2b02da16_isrForCrosstalkSources_1125902_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/d486716f-58cf-4a9a-85af-7a3e2b02da16_isrForCrosstalkSources_1125902_25.stderr parsl.dataflow.dflow DEBUG: Task 1806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1806 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1806 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1806 has memoization hash cebd163080828a1d733e870ce7eed928 parsl.dataflow.memoization INFO: Task 1806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aeea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1806 try 0 launched on executor multi with executor id 1807 parsl.dataflow.dflow INFO: Standard output for task 1806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e9db2b88-91f8-4b10-a11f-27ac4e8ad660_isrForCrosstalkSources_1125937_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e9db2b88-91f8-4b10-a11f-27ac4e8ad660_isrForCrosstalkSources_1125937_41.stderr parsl.dataflow.dflow DEBUG: Task 1807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1807 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1807 has memoization hash 286094600db550c18820328cb33dadf2 parsl.dataflow.memoization INFO: Task 1807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aeeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1807 try 0 launched on executor multi with executor id 1808 parsl.dataflow.dflow INFO: Standard output for task 1807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/4e16b24e-fd53-4d43-9ff3-015cfddc64ef_isrForCrosstalkSources_1125931_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/4e16b24e-fd53-4d43-9ff3-015cfddc64ef_isrForCrosstalkSources_1125931_2.stderr parsl.dataflow.dflow DEBUG: Task 1808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1808 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1808 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1808 has memoization hash cab8e390d81728f1e127571adbb2ba3e parsl.dataflow.memoization INFO: Task 1808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aeeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1808 try 0 launched on executor multi with executor id 1809 parsl.dataflow.dflow INFO: Standard output for task 1808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/5c8a5c04-e763-4476-8d16-7ad39d38eb57_isrForCrosstalkSources_1125939_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/5c8a5c04-e763-4476-8d16-7ad39d38eb57_isrForCrosstalkSources_1125939_31.stderr parsl.dataflow.dflow DEBUG: Task 1809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1809 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1809 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1809 has memoization hash 34ad81e4cbc0ff0492a7d528d05949f4 parsl.dataflow.memoization INFO: Task 1809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aeec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1809 try 0 launched on executor multi with executor id 1810 parsl.dataflow.dflow INFO: Standard output for task 1809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/57e3b4b8-4390-4d5e-9265-ac720bdc2419_isrForCrosstalkSources_1125902_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/57e3b4b8-4390-4d5e-9265-ac720bdc2419_isrForCrosstalkSources_1125902_60.stderr parsl.dataflow.dflow DEBUG: Task 1810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1810 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1810 has memoization hash b1639c7da6cf9f9db4ec93c03232a345 parsl.dataflow.memoization INFO: Task 1810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aeeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1810 try 0 launched on executor multi with executor id 1811 parsl.dataflow.dflow INFO: Standard output for task 1810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/e397940c-e2ac-4bbe-aeeb-0cef508cd0f0_isrForCrosstalkSources_1125923_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/e397940c-e2ac-4bbe-aeeb-0cef508cd0f0_isrForCrosstalkSources_1125923_60.stderr parsl.dataflow.dflow DEBUG: Task 1811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1811 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1811 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1811 has memoization hash e7ae1a890ba3ca9f72d545c711b993a7 parsl.dataflow.memoization INFO: Task 1811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aeed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1811 try 0 launched on executor multi with executor id 1812 parsl.dataflow.dflow INFO: Standard output for task 1811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/864472bf-81fb-48e5-a951-b6cbf0cd87d7_isrForCrosstalkSources_1125896_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/864472bf-81fb-48e5-a951-b6cbf0cd87d7_isrForCrosstalkSources_1125896_11.stderr parsl.dataflow.dflow DEBUG: Task 1812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1812 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1812 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1812 has memoization hash dcdf9814a9a246e7000817851ab86a8c parsl.dataflow.memoization INFO: Task 1812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aeee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1812 try 0 launched on executor multi with executor id 1813 parsl.dataflow.dflow INFO: Standard output for task 1812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/7443d8bc-64c1-4e67-ad76-fee859f21bcb_isrForCrosstalkSources_1125938_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/7443d8bc-64c1-4e67-ad76-fee859f21bcb_isrForCrosstalkSources_1125938_29.stderr parsl.dataflow.dflow DEBUG: Task 1813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1813 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1813 has memoization hash 57dfde9f752e36503c8978a29bdc318a parsl.dataflow.memoization INFO: Task 1813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aeede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1813 try 0 launched on executor multi with executor id 1814 parsl.dataflow.dflow INFO: Standard output for task 1813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/7a159298-6a5b-46ca-8d41-2d25825f78b6_isrForCrosstalkSources_1125912_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/7a159298-6a5b-46ca-8d41-2d25825f78b6_isrForCrosstalkSources_1125912_57.stderr parsl.dataflow.dflow DEBUG: Task 1814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1814 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1814 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1814 has memoization hash b3268b0bff7e4a7b9406017c8d787108 parsl.dataflow.memoization INFO: Task 1814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aeef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1814 try 0 launched on executor multi with executor id 1815 parsl.dataflow.dflow INFO: Standard output for task 1814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/c16b505c-5008-4ac4-8863-f63e7101d8d6_isrForCrosstalkSources_1125900_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/c16b505c-5008-4ac4-8863-f63e7101d8d6_isrForCrosstalkSources_1125900_3.stderr parsl.dataflow.dflow DEBUG: Task 1815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1815 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1815 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1815 has memoization hash e4085ec0f26848652f06658f044374c8 parsl.dataflow.memoization INFO: Task 1815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aeefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1815 try 0 launched on executor multi with executor id 1816 parsl.dataflow.dflow INFO: Standard output for task 1815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/032d1fe2-b430-4fdc-98ba-1a40c6b7b045_isrForCrosstalkSources_1125944_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/032d1fe2-b430-4fdc-98ba-1a40c6b7b045_isrForCrosstalkSources_1125944_22.stderr parsl.dataflow.dflow DEBUG: Task 1816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1816 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1816 has memoization hash ed011f48c5e0e1e3f9581d1d7605bca5 parsl.dataflow.memoization INFO: Task 1816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aef060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1816 try 0 launched on executor multi with executor id 1817 parsl.dataflow.dflow INFO: Standard output for task 1816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/9c6fb772-7f60-4a08-89f7-fddbfcda9ec4_isrForCrosstalkSources_1125940_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/9c6fb772-7f60-4a08-89f7-fddbfcda9ec4_isrForCrosstalkSources_1125940_25.stderr parsl.dataflow.dflow DEBUG: Task 1817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1817 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1817 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1817 has memoization hash aecf30e5d750fc2ba765a4480c6da825 parsl.dataflow.memoization INFO: Task 1817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aef1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1817 try 0 launched on executor multi with executor id 1818 parsl.dataflow.dflow INFO: Standard output for task 1817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/8a856cf7-be81-4c2e-81bb-7e7665e78d9a_isrForCrosstalkSources_1125937_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/8a856cf7-be81-4c2e-81bb-7e7665e78d9a_isrForCrosstalkSources_1125937_44.stderr parsl.dataflow.dflow DEBUG: Task 1818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1818 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1818 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1818 has memoization hash e0c58f143995b436039d8e22dc26fd64 parsl.dataflow.memoization INFO: Task 1818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aef100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1818 try 0 launched on executor multi with executor id 1819 parsl.dataflow.dflow INFO: Standard output for task 1818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/35743ccf-88af-4946-abfd-cd5b778a44f5_isrForCrosstalkSources_1125891_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/35743ccf-88af-4946-abfd-cd5b778a44f5_isrForCrosstalkSources_1125891_8.stderr parsl.dataflow.dflow DEBUG: Task 1819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1819 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1819 has memoization hash 8b41023694f8ceb4a26e12900a5a4c99 parsl.dataflow.memoization INFO: Task 1819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aef240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1819 try 0 launched on executor multi with executor id 1820 parsl.dataflow.dflow INFO: Standard output for task 1819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/650bd64b-b99f-4199-8ccc-143e7506c252_isrForCrosstalkSources_1125942_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/650bd64b-b99f-4199-8ccc-143e7506c252_isrForCrosstalkSources_1125942_1.stderr parsl.dataflow.dflow DEBUG: Task 1820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1820 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1820 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1820 has memoization hash 22c0a7042fb8442ce5b59f94bafb7e15 parsl.dataflow.memoization INFO: Task 1820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aef2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1820 try 0 launched on executor multi with executor id 1821 parsl.dataflow.dflow INFO: Standard output for task 1820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/694b6063-319f-4ca3-b340-95a2a3734a06_isrForCrosstalkSources_1125942_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/694b6063-319f-4ca3-b340-95a2a3734a06_isrForCrosstalkSources_1125942_60.stderr parsl.dataflow.dflow DEBUG: Task 1821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1821 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1821 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1821 has memoization hash 288e72258f8d526e55d1080a63ec0a1b parsl.dataflow.memoization INFO: Task 1821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aef380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1821 try 0 launched on executor multi with executor id 1822 parsl.dataflow.dflow INFO: Standard output for task 1821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/afb12a5a-d053-4859-a72b-eadb9fbe0c46_isrForCrosstalkSources_1125889_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/afb12a5a-d053-4859-a72b-eadb9fbe0c46_isrForCrosstalkSources_1125889_10.stderr parsl.dataflow.dflow DEBUG: Task 1822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1822 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1822 has memoization hash 4dd787eb526d97e83645f0367899e26c parsl.dataflow.memoization INFO: Task 1822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aef4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1822 try 0 launched on executor multi with executor id 1823 parsl.dataflow.dflow INFO: Standard output for task 1822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ffef047d-6248-4984-9e75-230298db02dc_isrForCrosstalkSources_1125947_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ffef047d-6248-4984-9e75-230298db02dc_isrForCrosstalkSources_1125947_31.stderr parsl.dataflow.dflow DEBUG: Task 1823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1823 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1823 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1823 has memoization hash e6870294a5524fe488ee725ec82aa7f1 parsl.dataflow.memoization INFO: Task 1823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aef420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1823 try 0 launched on executor multi with executor id 1824 parsl.dataflow.dflow INFO: Standard output for task 1823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/e970b819-165b-4d62-8668-771955b4e3f4_isrForCrosstalkSources_1125904_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/e970b819-165b-4d62-8668-771955b4e3f4_isrForCrosstalkSources_1125904_17.stderr parsl.dataflow.dflow DEBUG: Task 1824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1824 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1824 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1824 has memoization hash a31768b4dc581ae3efeccc435fd6c6bb parsl.dataflow.memoization INFO: Task 1824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aef560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1824 try 0 launched on executor multi with executor id 1825 parsl.dataflow.dflow INFO: Standard output for task 1824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/dec41bc0-b3df-4122-b0f4-283800d985f7_isrForCrosstalkSources_1125895_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/dec41bc0-b3df-4122-b0f4-283800d985f7_isrForCrosstalkSources_1125895_18.stderr parsl.dataflow.dflow DEBUG: Task 1825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1825 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1825 has memoization hash ff35ea2560beaddf403b48b6e01eb020 parsl.dataflow.memoization INFO: Task 1825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aef600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1825 try 0 launched on executor multi with executor id 1826 parsl.dataflow.dflow INFO: Standard output for task 1825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/84511bfb-c522-4512-942d-6190973e8693_isrForCrosstalkSources_1125896_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/84511bfb-c522-4512-942d-6190973e8693_isrForCrosstalkSources_1125896_47.stderr parsl.dataflow.dflow DEBUG: Task 1826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1826 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1826 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1826 has memoization hash 1c83ce9adf5f597584cc967b6594f58a parsl.dataflow.memoization INFO: Task 1826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aef6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1826 try 0 launched on executor multi with executor id 1827 parsl.dataflow.dflow INFO: Standard output for task 1826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f1e64d90-6e11-44f4-9720-7057e3c2a768_isrForCrosstalkSources_1125894_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f1e64d90-6e11-44f4-9720-7057e3c2a768_isrForCrosstalkSources_1125894_26.stderr parsl.dataflow.dflow DEBUG: Task 1827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1827 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1827 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1827 has memoization hash 8482e525e9d084bdebe2ae7955467a63 parsl.dataflow.memoization INFO: Task 1827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aef7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1827 try 0 launched on executor multi with executor id 1828 parsl.dataflow.dflow INFO: Standard output for task 1827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/a40598bb-a319-4d50-abf2-ec32b33ae49f_isrForCrosstalkSources_1125904_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/a40598bb-a319-4d50-abf2-ec32b33ae49f_isrForCrosstalkSources_1125904_39.stderr parsl.dataflow.dflow DEBUG: Task 1828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1828 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1828 has memoization hash 5c56cf63dff4aef35fa7133b2b9385a9 parsl.dataflow.memoization INFO: Task 1828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aef740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1828 try 0 launched on executor multi with executor id 1829 parsl.dataflow.dflow INFO: Standard output for task 1828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/48550974-1757-4058-b8b3-4567f20b2c79_isrForCrosstalkSources_1125945_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/48550974-1757-4058-b8b3-4567f20b2c79_isrForCrosstalkSources_1125945_38.stderr parsl.dataflow.dflow DEBUG: Task 1829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1829 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1829 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1829 has memoization hash 7ba0e942f24172c5fd2cca7a9505dca3 parsl.dataflow.memoization INFO: Task 1829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aef880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1829 try 0 launched on executor multi with executor id 1830 parsl.dataflow.dflow INFO: Standard output for task 1829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/2f37f18c-ef40-44e7-8c5c-33499aa4a80a_isrForCrosstalkSources_1125915_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/2f37f18c-ef40-44e7-8c5c-33499aa4a80a_isrForCrosstalkSources_1125915_62.stderr parsl.dataflow.dflow DEBUG: Task 1830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1830 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1830 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1830 has memoization hash f5b01765e404a440da30a9e526f840cd parsl.dataflow.memoization INFO: Task 1830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aef920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1830 try 0 launched on executor multi with executor id 1831 parsl.dataflow.dflow INFO: Standard output for task 1830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/892b508a-c1e5-4845-a43a-ac2efb488d03_isrForCrosstalkSources_1125936_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/892b508a-c1e5-4845-a43a-ac2efb488d03_isrForCrosstalkSources_1125936_12.stderr parsl.dataflow.dflow DEBUG: Task 1831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1831 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1831 has memoization hash eef30d20f9bd9cd112857c846e3db8c6 parsl.dataflow.memoization INFO: Task 1831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aef9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1831 try 0 launched on executor multi with executor id 1832 parsl.dataflow.dflow INFO: Standard output for task 1831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/5824174b-7b6c-41ed-9faa-ff50c00b3fd2_isrForCrosstalkSources_1125945_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/5824174b-7b6c-41ed-9faa-ff50c00b3fd2_isrForCrosstalkSources_1125945_5.stderr parsl.dataflow.dflow DEBUG: Task 1832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1832 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1832 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1832 has memoization hash 1070a1637a70be24cd84e504e3d72acb parsl.dataflow.memoization INFO: Task 1832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aefb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1832 try 0 launched on executor multi with executor id 1833 parsl.dataflow.dflow INFO: Standard output for task 1832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/ea81b2c9-a834-4a79-a0c6-f9a4346f3b4f_isrForCrosstalkSources_1125927_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/ea81b2c9-a834-4a79-a0c6-f9a4346f3b4f_isrForCrosstalkSources_1125927_49.stderr parsl.dataflow.dflow DEBUG: Task 1833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1833 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1833 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1833 has memoization hash c34b84fe163d224325cf8528cb8abee7 parsl.dataflow.memoization INFO: Task 1833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aefa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1833 try 0 launched on executor multi with executor id 1834 parsl.dataflow.dflow INFO: Standard output for task 1833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/74f27a1f-fb27-4e34-ad70-7f7d10939d97_isrForCrosstalkSources_1125935_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/74f27a1f-fb27-4e34-ad70-7f7d10939d97_isrForCrosstalkSources_1125935_4.stderr parsl.dataflow.dflow DEBUG: Task 1834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1834 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1834 has memoization hash 7f9f5f8c9f4236b54ad3538f98b9f253 parsl.dataflow.memoization INFO: Task 1834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aefba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1834 try 0 launched on executor multi with executor id 1835 parsl.dataflow.dflow INFO: Standard output for task 1834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/955efb03-8de2-42f9-b4d4-0b96fa087768_isrForCrosstalkSources_1125939_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/955efb03-8de2-42f9-b4d4-0b96fa087768_isrForCrosstalkSources_1125939_42.stderr parsl.dataflow.dflow DEBUG: Task 1835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1835 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1835 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1835 has memoization hash 6cddf9811731e2fa3b03cec3e7af6baf parsl.dataflow.memoization INFO: Task 1835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aefc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1835 try 0 launched on executor multi with executor id 1836 parsl.dataflow.dflow INFO: Standard output for task 1835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/badd2abe-0625-4aff-80fb-a0ef72cb9731_isrForCrosstalkSources_1125901_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/badd2abe-0625-4aff-80fb-a0ef72cb9731_isrForCrosstalkSources_1125901_38.stderr parsl.dataflow.dflow DEBUG: Task 1836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1836 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1836 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1836 has memoization hash e87318e09acb09869f6206495c3c9e1e parsl.dataflow.memoization INFO: Task 1836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aefce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1836 try 0 launched on executor multi with executor id 1837 parsl.dataflow.dflow INFO: Standard output for task 1836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/670abf8e-e874-49ba-aa29-ac2bbbebc16b_isrForCrosstalkSources_1125920_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/670abf8e-e874-49ba-aa29-ac2bbbebc16b_isrForCrosstalkSources_1125920_32.stderr parsl.dataflow.dflow DEBUG: Task 1837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1837 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1837 has memoization hash 4b93876efb66006e5533a03bec8015e8 parsl.dataflow.memoization INFO: Task 1837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aefe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1837 try 0 launched on executor multi with executor id 1838 parsl.dataflow.dflow INFO: Standard output for task 1837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/c895728a-a646-4235-b42b-2476ca9d1d7f_isrForCrosstalkSources_1125931_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/c895728a-a646-4235-b42b-2476ca9d1d7f_isrForCrosstalkSources_1125931_14.stderr parsl.dataflow.dflow DEBUG: Task 1838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1838 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1838 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1838 has memoization hash 534bd863ff762c50ffdeda4fb33b1c7c parsl.dataflow.memoization INFO: Task 1838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aefd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1838 try 0 launched on executor multi with executor id 1839 parsl.dataflow.dflow INFO: Standard output for task 1838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/6c68799e-bd19-47f0-a5e0-2d0979d8cfbf_isrForCrosstalkSources_1125936_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/6c68799e-bd19-47f0-a5e0-2d0979d8cfbf_isrForCrosstalkSources_1125936_40.stderr parsl.dataflow.dflow DEBUG: Task 1839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1839 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1839 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1839 has memoization hash edec7531f1f60d10bac81b8120b80806 parsl.dataflow.memoization INFO: Task 1839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aefec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1839 try 0 launched on executor multi with executor id 1840 parsl.dataflow.dflow INFO: Standard output for task 1839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/9cac976d-b217-4a10-8ed5-2f165128deb6_isrForCrosstalkSources_1125917_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/9cac976d-b217-4a10-8ed5-2f165128deb6_isrForCrosstalkSources_1125917_29.stderr parsl.dataflow.dflow DEBUG: Task 1840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1840 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1840 has memoization hash a69906461e191df6ecfd22682b0eb3b2 parsl.dataflow.memoization INFO: Task 1840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aeff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1840 try 0 launched on executor multi with executor id 1841 parsl.dataflow.dflow INFO: Standard output for task 1840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/fa7cc0d2-05d2-40b5-b787-21ea29f62977_isrForCrosstalkSources_1125944_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/fa7cc0d2-05d2-40b5-b787-21ea29f62977_isrForCrosstalkSources_1125944_5.stderr parsl.dataflow.dflow DEBUG: Task 1841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1841 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1841 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1841 has memoization hash 44fa9b26427f4f37977699120ff4e0ff parsl.dataflow.memoization INFO: Task 1841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39aec040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1841 try 0 launched on executor multi with executor id 1842 parsl.dataflow.dflow INFO: Standard output for task 1841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/156cc795-55fd-438d-aec4-7a3d5ac872f1_isrForCrosstalkSources_1125908_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/156cc795-55fd-438d-aec4-7a3d5ac872f1_isrForCrosstalkSources_1125908_38.stderr parsl.dataflow.dflow DEBUG: Task 1842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1842 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1842 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1842 has memoization hash fb802d7fe9efbc14aacf02a460f755b6 parsl.dataflow.memoization INFO: Task 1842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1842 try 0 launched on executor multi with executor id 1843 parsl.dataflow.dflow INFO: Standard output for task 1842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/3602efc7-a284-498a-879e-1debf9012305_isrForCrosstalkSources_1125946_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/3602efc7-a284-498a-879e-1debf9012305_isrForCrosstalkSources_1125946_16.stderr parsl.dataflow.dflow DEBUG: Task 1843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1843 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1843 has memoization hash e79d5698cb15ffbd2eda38259a77c16c parsl.dataflow.memoization INFO: Task 1843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1843 try 0 launched on executor multi with executor id 1844 parsl.dataflow.dflow INFO: Standard output for task 1843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/0c9a12eb-4af7-488b-84a3-3a5609b90b0e_isrForCrosstalkSources_1125931_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/0c9a12eb-4af7-488b-84a3-3a5609b90b0e_isrForCrosstalkSources_1125931_51.stderr parsl.dataflow.dflow DEBUG: Task 1844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1844 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1844 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1844 has memoization hash 95f68bbff2c5e143e3e4c330ae169748 parsl.dataflow.memoization INFO: Task 1844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1844 try 0 launched on executor multi with executor id 1845 parsl.dataflow.dflow INFO: Standard output for task 1844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/34e00344-4513-4591-8448-4080723802d5_isrForCrosstalkSources_1125925_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/34e00344-4513-4591-8448-4080723802d5_isrForCrosstalkSources_1125925_23.stderr parsl.dataflow.dflow DEBUG: Task 1845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1845 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1845 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1845 has memoization hash b8e32a3bbf4996c6c24d494544ada135 parsl.dataflow.memoization INFO: Task 1845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1845 try 0 launched on executor multi with executor id 1846 parsl.dataflow.dflow INFO: Standard output for task 1845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/6cbb2b52-5fc9-4381-9a00-d66716383348_isrForCrosstalkSources_1125908_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/6cbb2b52-5fc9-4381-9a00-d66716383348_isrForCrosstalkSources_1125908_59.stderr parsl.dataflow.dflow DEBUG: Task 1846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1846 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1846 has memoization hash 4c013c3f6fad8af934cf79811ab4a7fc parsl.dataflow.memoization INFO: Task 1846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1846 try 0 launched on executor multi with executor id 1847 parsl.dataflow.dflow INFO: Standard output for task 1846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/af1d8b5d-ca49-4d6c-8c21-28a273bc23e0_isrForCrosstalkSources_1125896_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/af1d8b5d-ca49-4d6c-8c21-28a273bc23e0_isrForCrosstalkSources_1125896_20.stderr parsl.dataflow.dflow DEBUG: Task 1847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1847 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1847 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1847 has memoization hash 7f4c543a19d8e202430e37e1429d9fe5 parsl.dataflow.memoization INFO: Task 1847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1847 try 0 launched on executor multi with executor id 1848 parsl.dataflow.dflow INFO: Standard output for task 1847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/0e1507ce-3b96-48df-b992-22366a856870_isrForCrosstalkSources_1125926_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/0e1507ce-3b96-48df-b992-22366a856870_isrForCrosstalkSources_1125926_47.stderr parsl.dataflow.dflow DEBUG: Task 1848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1848 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1848 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1848 has memoization hash 65dfa2c9c16f4f51f5d04cedf5410968 parsl.dataflow.memoization INFO: Task 1848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1848 try 0 launched on executor multi with executor id 1849 parsl.dataflow.dflow INFO: Standard output for task 1848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/dbadbf9c-2d18-4b64-8839-91629bd4b417_isrForCrosstalkSources_1125932_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/dbadbf9c-2d18-4b64-8839-91629bd4b417_isrForCrosstalkSources_1125932_19.stderr parsl.dataflow.dflow DEBUG: Task 1849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1849 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1849 has memoization hash b7de5a50724a867f3bf6e2b86b4dba62 parsl.dataflow.memoization INFO: Task 1849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1849 try 0 launched on executor multi with executor id 1850 parsl.dataflow.dflow INFO: Standard output for task 1849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/39dc5fa9-98d8-47cb-8624-aece02a25108_isrForCrosstalkSources_1125938_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/39dc5fa9-98d8-47cb-8624-aece02a25108_isrForCrosstalkSources_1125938_38.stderr parsl.dataflow.dflow DEBUG: Task 1850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1850 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1850 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1850 has memoization hash bb03f102cded56735e235a9669496448 parsl.dataflow.memoization INFO: Task 1850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1850 try 0 launched on executor multi with executor id 1851 parsl.dataflow.dflow INFO: Standard output for task 1850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/ecf61a15-df88-46f5-8e5b-2bf913251787_isrForCrosstalkSources_1125908_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/ecf61a15-df88-46f5-8e5b-2bf913251787_isrForCrosstalkSources_1125908_21.stderr parsl.dataflow.dflow DEBUG: Task 1851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1851 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1851 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1851 has memoization hash a11dd403200d1d9397985fe29498b0c8 parsl.dataflow.memoization INFO: Task 1851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1851 try 0 launched on executor multi with executor id 1852 parsl.dataflow.dflow INFO: Standard output for task 1851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/ae49fb2d-8ca5-43d1-ac28-ff44150e50bd_isrForCrosstalkSources_1125895_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/ae49fb2d-8ca5-43d1-ac28-ff44150e50bd_isrForCrosstalkSources_1125895_38.stderr parsl.dataflow.dflow DEBUG: Task 1852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1852 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1852 has memoization hash 7d0e520cfb16fcc5b7b8e86c885d3305 parsl.dataflow.memoization INFO: Task 1852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1852 try 0 launched on executor multi with executor id 1853 parsl.dataflow.dflow INFO: Standard output for task 1852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/4bbc8ec1-ad1d-41d5-a8c9-5642c4a140b4_isrForCrosstalkSources_1125923_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/4bbc8ec1-ad1d-41d5-a8c9-5642c4a140b4_isrForCrosstalkSources_1125923_24.stderr parsl.dataflow.dflow DEBUG: Task 1853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1853 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1853 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1853 has memoization hash 7846e43d187ab559108d66984c005b13 parsl.dataflow.memoization INFO: Task 1853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1853 try 0 launched on executor multi with executor id 1854 parsl.dataflow.dflow INFO: Standard output for task 1853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/53c4768b-beb6-44a0-ab67-e31e7f4a0a5d_isrForCrosstalkSources_1125946_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/53c4768b-beb6-44a0-ab67-e31e7f4a0a5d_isrForCrosstalkSources_1125946_54.stderr parsl.dataflow.dflow DEBUG: Task 1854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1854 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1854 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1854 has memoization hash 40a395944c73d4c70c83677f134e112e parsl.dataflow.memoization INFO: Task 1854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1854 try 0 launched on executor multi with executor id 1855 parsl.dataflow.dflow INFO: Standard output for task 1854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/431e2cd7-b183-486f-87ca-263e953e8bed_isrForCrosstalkSources_1125910_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/431e2cd7-b183-486f-87ca-263e953e8bed_isrForCrosstalkSources_1125910_47.stderr parsl.dataflow.dflow DEBUG: Task 1855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1855 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1855 has memoization hash 2ec73759558def9c88c47879727faebe parsl.dataflow.memoization INFO: Task 1855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1855 try 0 launched on executor multi with executor id 1856 parsl.dataflow.dflow INFO: Standard output for task 1855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/0f86e093-b6a0-4eb8-9e9b-72e7f55eb9b6_isrForCrosstalkSources_1125920_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/0f86e093-b6a0-4eb8-9e9b-72e7f55eb9b6_isrForCrosstalkSources_1125920_53.stderr parsl.dataflow.dflow DEBUG: Task 1856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1856 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1856 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1856 has memoization hash f988dfc007dd86bbc5d84d82bf42e613 parsl.dataflow.memoization INFO: Task 1856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1856 try 0 launched on executor multi with executor id 1857 parsl.dataflow.dflow INFO: Standard output for task 1856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9654a7d4-48ae-49bc-94fc-fee923ef6175_isrForCrosstalkSources_1125921_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9654a7d4-48ae-49bc-94fc-fee923ef6175_isrForCrosstalkSources_1125921_45.stderr parsl.dataflow.dflow DEBUG: Task 1857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1857 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1857 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1857 has memoization hash 1ced38e9c79c8d9c26ea457272a306c9 parsl.dataflow.memoization INFO: Task 1857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1857 try 0 launched on executor multi with executor id 1858 parsl.dataflow.dflow INFO: Standard output for task 1857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/621aa51e-5dda-46bb-bb6f-48a59bafd98b_isrForCrosstalkSources_1125944_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/621aa51e-5dda-46bb-bb6f-48a59bafd98b_isrForCrosstalkSources_1125944_31.stderr parsl.dataflow.dflow DEBUG: Task 1858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1858 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1858 has memoization hash 97d459189abc0b8e91361fe679d9c582 parsl.dataflow.memoization INFO: Task 1858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1858 try 0 launched on executor multi with executor id 1859 parsl.dataflow.dflow INFO: Standard output for task 1858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/c36309e8-d5c0-4428-9201-7f62d6914e0c_isrForCrosstalkSources_1125908_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/c36309e8-d5c0-4428-9201-7f62d6914e0c_isrForCrosstalkSources_1125908_15.stderr parsl.dataflow.dflow DEBUG: Task 1859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1859 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1859 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1859 has memoization hash 99d795608abfd5978fa61d3b50ee55f7 parsl.dataflow.memoization INFO: Task 1859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1859 try 0 launched on executor multi with executor id 1860 parsl.dataflow.dflow INFO: Standard output for task 1859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/1210ab85-128e-478b-a120-5d5074fbb322_isrForCrosstalkSources_1125888_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/1210ab85-128e-478b-a120-5d5074fbb322_isrForCrosstalkSources_1125888_9.stderr parsl.dataflow.dflow DEBUG: Task 1860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1860 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1860 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1860 has memoization hash d3e32b50c04c0174be0c849ee3ebdbf1 parsl.dataflow.memoization INFO: Task 1860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1860 try 0 launched on executor multi with executor id 1861 parsl.dataflow.dflow INFO: Standard output for task 1860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/6d1f169e-4307-480f-a009-6fadd3ec0eed_isrForCrosstalkSources_1125897_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/6d1f169e-4307-480f-a009-6fadd3ec0eed_isrForCrosstalkSources_1125897_35.stderr parsl.dataflow.dflow DEBUG: Task 1861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1861 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1861 has memoization hash 1be3f20431f5fa26102acd3932828f79 parsl.dataflow.memoization INFO: Task 1861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1861 try 0 launched on executor multi with executor id 1862 parsl.dataflow.dflow INFO: Standard output for task 1861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4dcc04d6-0202-40c4-9f95-e2e87e93294e_isrForCrosstalkSources_1125908_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4dcc04d6-0202-40c4-9f95-e2e87e93294e_isrForCrosstalkSources_1125908_24.stderr parsl.dataflow.dflow DEBUG: Task 1862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1862 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1862 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1862 has memoization hash a2e6bfbaab490e5a2922c3863356de77 parsl.dataflow.memoization INFO: Task 1862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1862 try 0 launched on executor multi with executor id 1863 parsl.dataflow.dflow INFO: Standard output for task 1862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/f4c6159e-2c84-4e91-a192-9b5e04183ac0_isrForCrosstalkSources_1125925_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/f4c6159e-2c84-4e91-a192-9b5e04183ac0_isrForCrosstalkSources_1125925_56.stderr parsl.dataflow.dflow DEBUG: Task 1863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1863 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1863 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1863 has memoization hash 881ba9579d94d3f6e036b9f85cb00208 parsl.dataflow.memoization INFO: Task 1863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1863 try 0 launched on executor multi with executor id 1864 parsl.dataflow.dflow INFO: Standard output for task 1863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/b080f8c3-b4ee-4809-a15a-be48db02ef11_isrForCrosstalkSources_1125930_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/b080f8c3-b4ee-4809-a15a-be48db02ef11_isrForCrosstalkSources_1125930_58.stderr parsl.dataflow.dflow DEBUG: Task 1864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1864 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1864 has memoization hash 1088521c8cde1e347b0eadb97fc935d3 parsl.dataflow.memoization INFO: Task 1864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1864 try 0 launched on executor multi with executor id 1865 parsl.dataflow.dflow INFO: Standard output for task 1864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/126751ff-2bcb-449e-9e45-60fa5abf0a96_isrForCrosstalkSources_1125911_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/126751ff-2bcb-449e-9e45-60fa5abf0a96_isrForCrosstalkSources_1125911_28.stderr parsl.dataflow.dflow DEBUG: Task 1865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1865 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1865 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1865 has memoization hash aadc0173411415d6059d2948a825e1c4 parsl.dataflow.memoization INFO: Task 1865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1865 try 0 launched on executor multi with executor id 1866 parsl.dataflow.dflow INFO: Standard output for task 1865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/f731082d-e620-4148-841b-ad979df5050e_isrForCrosstalkSources_1125936_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/f731082d-e620-4148-841b-ad979df5050e_isrForCrosstalkSources_1125936_48.stderr parsl.dataflow.dflow DEBUG: Task 1866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1866 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1866 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1866 has memoization hash d1235dd019750e0c1320e3f4a63bcddb parsl.dataflow.memoization INFO: Task 1866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1866 try 0 launched on executor multi with executor id 1867 parsl.dataflow.dflow INFO: Standard output for task 1866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/3c4eaa79-986b-453b-a4bb-338a267965a7_isrForCrosstalkSources_1125895_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/3c4eaa79-986b-453b-a4bb-338a267965a7_isrForCrosstalkSources_1125895_16.stderr parsl.dataflow.dflow DEBUG: Task 1867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1867 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1867 has memoization hash 08bdea1e6f70238821313168eeea8fb7 parsl.dataflow.memoization INFO: Task 1867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1867 try 0 launched on executor multi with executor id 1868 parsl.dataflow.dflow INFO: Standard output for task 1867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a87b5452-2885-45c2-a17b-43e620e7e158_isrForCrosstalkSources_1125938_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a87b5452-2885-45c2-a17b-43e620e7e158_isrForCrosstalkSources_1125938_23.stderr parsl.dataflow.dflow DEBUG: Task 1868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1868 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1868 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1868 has memoization hash b3a69aeecd33574f20b27185ee635c2a parsl.dataflow.memoization INFO: Task 1868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1868 try 0 launched on executor multi with executor id 1869 parsl.dataflow.dflow INFO: Standard output for task 1868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/40e3cc81-274c-4ed1-b77e-28df8aef78fe_isrForCrosstalkSources_1125927_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/40e3cc81-274c-4ed1-b77e-28df8aef78fe_isrForCrosstalkSources_1125927_29.stderr parsl.dataflow.dflow DEBUG: Task 1869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1869 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1869 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1869 has memoization hash a0cec5727a658bd9ee2c74b9dd7f6977 parsl.dataflow.memoization INFO: Task 1869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1869 try 0 launched on executor multi with executor id 1870 parsl.dataflow.dflow INFO: Standard output for task 1869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/2fec0063-3851-4ddb-9be0-ad5a2e7280b4_isrForCrosstalkSources_1125912_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/2fec0063-3851-4ddb-9be0-ad5a2e7280b4_isrForCrosstalkSources_1125912_16.stderr parsl.dataflow.dflow DEBUG: Task 1870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1870 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1870 has memoization hash 87ba3aa769f6c16aacced27abd2c40d2 parsl.dataflow.memoization INFO: Task 1870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1870 try 0 launched on executor multi with executor id 1871 parsl.dataflow.dflow INFO: Standard output for task 1870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/7108011b-f11c-4e25-9eab-0fe9d00b9e54_isrForCrosstalkSources_1125936_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/7108011b-f11c-4e25-9eab-0fe9d00b9e54_isrForCrosstalkSources_1125936_60.stderr parsl.dataflow.dflow DEBUG: Task 1871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1871 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1871 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1871 has memoization hash 03aba18fa49c8e58e3d8d9fa39c3c7d0 parsl.dataflow.memoization INFO: Task 1871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1871 try 0 launched on executor multi with executor id 1872 parsl.dataflow.dflow INFO: Standard output for task 1871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/ad84d27d-5e9b-42a5-9478-6985840f1a46_isrForCrosstalkSources_1125899_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/ad84d27d-5e9b-42a5-9478-6985840f1a46_isrForCrosstalkSources_1125899_19.stderr parsl.dataflow.dflow DEBUG: Task 1872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1872 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1872 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1872 has memoization hash 4dc734363f9a3eaf5e3d262e1ddd9c2c parsl.dataflow.memoization INFO: Task 1872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1872 try 0 launched on executor multi with executor id 1873 parsl.dataflow.dflow INFO: Standard output for task 1872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f6e486d5-6750-4b54-95e9-d3ab08fcd4bf_isrForCrosstalkSources_1125897_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f6e486d5-6750-4b54-95e9-d3ab08fcd4bf_isrForCrosstalkSources_1125897_33.stderr parsl.dataflow.dflow DEBUG: Task 1873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1873 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1873 has memoization hash cabbe2c05370ad15bf84bfcbef8d1dad parsl.dataflow.memoization INFO: Task 1873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1873 try 0 launched on executor multi with executor id 1874 parsl.dataflow.dflow INFO: Standard output for task 1873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f19d9da3-94c5-4194-ad4c-7576f8c8a2c0_isrForCrosstalkSources_1125930_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f19d9da3-94c5-4194-ad4c-7576f8c8a2c0_isrForCrosstalkSources_1125930_21.stderr parsl.dataflow.dflow DEBUG: Task 1874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1874 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1874 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1874 has memoization hash c1e4192b149f5f7d3379245df9115bc5 parsl.dataflow.memoization INFO: Task 1874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1874 try 0 launched on executor multi with executor id 1875 parsl.dataflow.dflow INFO: Standard output for task 1874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/49a66dd1-95f1-4398-9f88-93bcf3dd72d0_isrForCrosstalkSources_1125942_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/49a66dd1-95f1-4398-9f88-93bcf3dd72d0_isrForCrosstalkSources_1125942_10.stderr parsl.dataflow.dflow DEBUG: Task 1875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1875 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1875 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1875 has memoization hash 3f07e965e33829fe81c9e03c3f53d30a parsl.dataflow.memoization INFO: Task 1875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1875 try 0 launched on executor multi with executor id 1876 parsl.dataflow.dflow INFO: Standard output for task 1875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3b741490-83d9-426a-aa89-356f3a5c9c44_isrForCrosstalkSources_1125920_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3b741490-83d9-426a-aa89-356f3a5c9c44_isrForCrosstalkSources_1125920_30.stderr parsl.dataflow.dflow DEBUG: Task 1876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1876 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1876 has memoization hash 385174f1d8032867b56f4d44b7014158 parsl.dataflow.memoization INFO: Task 1876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1876 try 0 launched on executor multi with executor id 1877 parsl.dataflow.dflow INFO: Standard output for task 1876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/9f8a10ef-083f-4a64-ab7e-67747cfcdc5b_isrForCrosstalkSources_1125923_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/9f8a10ef-083f-4a64-ab7e-67747cfcdc5b_isrForCrosstalkSources_1125923_39.stderr parsl.dataflow.dflow DEBUG: Task 1877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1877 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1877 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1877 has memoization hash 9fdf4f73255e6dcbcc4c0fcb9c4094dc parsl.dataflow.memoization INFO: Task 1877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1877 try 0 launched on executor multi with executor id 1878 parsl.dataflow.dflow INFO: Standard output for task 1877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/f31c1848-dfbb-4d42-a786-55a6eda7855e_isrForCrosstalkSources_1125920_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/f31c1848-dfbb-4d42-a786-55a6eda7855e_isrForCrosstalkSources_1125920_17.stderr parsl.dataflow.dflow DEBUG: Task 1878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1878 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1878 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1878 has memoization hash f8001abcc83cb1a251eda50469ff2a24 parsl.dataflow.memoization INFO: Task 1878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1878 try 0 launched on executor multi with executor id 1879 parsl.dataflow.dflow INFO: Standard output for task 1878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/b2c27efc-68df-4575-9239-70b9e6e983d8_isrForCrosstalkSources_1125929_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/b2c27efc-68df-4575-9239-70b9e6e983d8_isrForCrosstalkSources_1125929_52.stderr parsl.dataflow.dflow DEBUG: Task 1879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1879 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1879 has memoization hash 56a38ad1e3710725cfc8963a706df44a parsl.dataflow.memoization INFO: Task 1879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1879 try 0 launched on executor multi with executor id 1880 parsl.dataflow.dflow INFO: Standard output for task 1879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/752ddc0a-3bec-472f-9263-9e2dffe6d617_isrForCrosstalkSources_1125943_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/752ddc0a-3bec-472f-9263-9e2dffe6d617_isrForCrosstalkSources_1125943_55.stderr parsl.dataflow.dflow DEBUG: Task 1880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1880 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1880 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1880 has memoization hash 59a92c41e895131a12362194685b4b9e parsl.dataflow.memoization INFO: Task 1880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1880 try 0 launched on executor multi with executor id 1881 parsl.dataflow.dflow INFO: Standard output for task 1880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/ce3c9462-4633-4763-b581-337b020fb462_isrForCrosstalkSources_1125931_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/ce3c9462-4633-4763-b581-337b020fb462_isrForCrosstalkSources_1125931_32.stderr parsl.dataflow.dflow DEBUG: Task 1881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1881 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1881 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1881 has memoization hash d559fdd0b1c7d43aea4ef0bc94619f3a parsl.dataflow.memoization INFO: Task 1881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1881 try 0 launched on executor multi with executor id 1882 parsl.dataflow.dflow INFO: Standard output for task 1881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/62096c0a-41c3-4fb3-8ee9-c82aac219202_isrForCrosstalkSources_1125921_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/62096c0a-41c3-4fb3-8ee9-c82aac219202_isrForCrosstalkSources_1125921_52.stderr parsl.dataflow.dflow DEBUG: Task 1882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1882 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1882 has memoization hash d055dc2f172c227b875790f498d72789 parsl.dataflow.memoization INFO: Task 1882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1882 try 0 launched on executor multi with executor id 1883 parsl.dataflow.dflow INFO: Standard output for task 1882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/cded9999-9913-4d80-b6b3-7a5932a9c386_isrForCrosstalkSources_1125914_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/cded9999-9913-4d80-b6b3-7a5932a9c386_isrForCrosstalkSources_1125914_60.stderr parsl.dataflow.dflow DEBUG: Task 1883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1883 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1883 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1883 has memoization hash 57b5ef994f9e3bd797a7b544ef596a2c parsl.dataflow.memoization INFO: Task 1883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1883 try 0 launched on executor multi with executor id 1884 parsl.dataflow.dflow INFO: Standard output for task 1883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/54f6e5fa-c490-4f7c-be9c-c0399d6a13c4_isrForCrosstalkSources_1125943_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/54f6e5fa-c490-4f7c-be9c-c0399d6a13c4_isrForCrosstalkSources_1125943_15.stderr parsl.dataflow.dflow DEBUG: Task 1884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1884 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1884 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1884 has memoization hash 6787da549ff5c681ed8fa210a38fe081 parsl.dataflow.memoization INFO: Task 1884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1884 try 0 launched on executor multi with executor id 1885 parsl.dataflow.dflow INFO: Standard output for task 1884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/92fe8a36-5e86-4164-98c3-34d2e9424478_isrForCrosstalkSources_1125910_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/92fe8a36-5e86-4164-98c3-34d2e9424478_isrForCrosstalkSources_1125910_60.stderr parsl.dataflow.dflow DEBUG: Task 1885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1885 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1885 has memoization hash 97a680f5d8fe5adf1fbbe7447744672f parsl.dataflow.memoization INFO: Task 1885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1885 try 0 launched on executor multi with executor id 1886 parsl.dataflow.dflow INFO: Standard output for task 1885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/a59d7eba-e17d-408f-91c8-4dbb962ccb1b_isrForCrosstalkSources_1125941_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/a59d7eba-e17d-408f-91c8-4dbb962ccb1b_isrForCrosstalkSources_1125941_19.stderr parsl.dataflow.dflow DEBUG: Task 1886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1886 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1886 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1886 has memoization hash 6cddcd6019b988b3cba72299e8289839 parsl.dataflow.memoization INFO: Task 1886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1886 try 0 launched on executor multi with executor id 1887 parsl.dataflow.dflow INFO: Standard output for task 1886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/d0f85e85-2a0e-4306-b63a-b550e4ff375b_isrForCrosstalkSources_1125921_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/d0f85e85-2a0e-4306-b63a-b550e4ff375b_isrForCrosstalkSources_1125921_55.stderr parsl.dataflow.dflow DEBUG: Task 1887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1887 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1887 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1887 has memoization hash 999186acd41a0c09e3ab3328c82e3483 parsl.dataflow.memoization INFO: Task 1887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1887 try 0 launched on executor multi with executor id 1888 parsl.dataflow.dflow INFO: Standard output for task 1887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/fa547d91-ce4a-4736-b11b-11869878d5d3_isrForCrosstalkSources_1125914_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/fa547d91-ce4a-4736-b11b-11869878d5d3_isrForCrosstalkSources_1125914_58.stderr parsl.dataflow.dflow DEBUG: Task 1888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1888 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1888 has memoization hash e1e1b1baab1a9caddc6adc6341f1e38c parsl.dataflow.memoization INFO: Task 1888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1888 try 0 launched on executor multi with executor id 1889 parsl.dataflow.dflow INFO: Standard output for task 1888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/ed877f50-5909-49b7-99ac-b4e83b2821d1_isrForCrosstalkSources_1125891_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/ed877f50-5909-49b7-99ac-b4e83b2821d1_isrForCrosstalkSources_1125891_5.stderr parsl.dataflow.dflow DEBUG: Task 1889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1889 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1889 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1889 has memoization hash bb1f78c445e25f85be199c4ee5acdbfc parsl.dataflow.memoization INFO: Task 1889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1889 try 0 launched on executor multi with executor id 1890 parsl.dataflow.dflow INFO: Standard output for task 1889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/63b2ff4f-05c9-4356-96f5-7174a4a3f4fe_isrForCrosstalkSources_1125930_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/63b2ff4f-05c9-4356-96f5-7174a4a3f4fe_isrForCrosstalkSources_1125930_41.stderr parsl.dataflow.dflow DEBUG: Task 1890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1890 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1890 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1890 has memoization hash 04514ec5dfaf93d880398826860f4bee parsl.dataflow.memoization INFO: Task 1890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1890 try 0 launched on executor multi with executor id 1891 parsl.dataflow.dflow INFO: Standard output for task 1890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/8910b4c3-2f57-4fdd-a767-ee2d31e810bd_isrForCrosstalkSources_1125909_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/8910b4c3-2f57-4fdd-a767-ee2d31e810bd_isrForCrosstalkSources_1125909_49.stderr parsl.dataflow.dflow DEBUG: Task 1891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1891 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1891 has memoization hash fbe46a946bb6d5efc44dcdadd3688d8c parsl.dataflow.memoization INFO: Task 1891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1891 try 0 launched on executor multi with executor id 1892 parsl.dataflow.dflow INFO: Standard output for task 1891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/b3380d55-31d0-4396-8f8a-3aba126952ec_isrForCrosstalkSources_1125888_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/b3380d55-31d0-4396-8f8a-3aba126952ec_isrForCrosstalkSources_1125888_24.stderr parsl.dataflow.dflow DEBUG: Task 1892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1892 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1892 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1892 has memoization hash d648a879ff8a02d6541d1c1cf31fcc8e parsl.dataflow.memoization INFO: Task 1892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1892 try 0 launched on executor multi with executor id 1893 parsl.dataflow.dflow INFO: Standard output for task 1892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b52956ec-2106-4fb2-bb37-480996f5320b_isrForCrosstalkSources_1125938_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b52956ec-2106-4fb2-bb37-480996f5320b_isrForCrosstalkSources_1125938_39.stderr parsl.dataflow.dflow DEBUG: Task 1893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1893 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1893 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1893 has memoization hash fa4ed3873adcb0f51e8b008920f706e5 parsl.dataflow.memoization INFO: Task 1893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1893 try 0 launched on executor multi with executor id 1894 parsl.dataflow.dflow INFO: Standard output for task 1893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/bb032897-2bcb-4957-9f83-feca7ec0e4e4_isrForCrosstalkSources_1125905_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/bb032897-2bcb-4957-9f83-feca7ec0e4e4_isrForCrosstalkSources_1125905_5.stderr parsl.dataflow.dflow DEBUG: Task 1894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1894 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1894 has memoization hash 6ebc1f063f7a1f93bb1e575e9ab6d788 parsl.dataflow.memoization INFO: Task 1894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1894 try 0 launched on executor multi with executor id 1895 parsl.dataflow.dflow INFO: Standard output for task 1894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/74d90df9-1feb-4051-a2d0-263dc781ed2b_isrForCrosstalkSources_1125917_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/74d90df9-1feb-4051-a2d0-263dc781ed2b_isrForCrosstalkSources_1125917_5.stderr parsl.dataflow.dflow DEBUG: Task 1895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1895 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1895 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1895 has memoization hash 467f5c2d2b1495407ab6b2217b508bd8 parsl.dataflow.memoization INFO: Task 1895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1895 try 0 launched on executor multi with executor id 1896 parsl.dataflow.dflow INFO: Standard output for task 1895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/b7c32181-713a-45fc-b30b-d1adcd507fc4_isrForCrosstalkSources_1125946_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/b7c32181-713a-45fc-b30b-d1adcd507fc4_isrForCrosstalkSources_1125946_9.stderr parsl.dataflow.dflow DEBUG: Task 1896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1896 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1896 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1896 has memoization hash bc935401b176a128ebba32a4c488cf75 parsl.dataflow.memoization INFO: Task 1896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1896 try 0 launched on executor multi with executor id 1897 parsl.dataflow.dflow INFO: Standard output for task 1896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/3123e2cb-0017-4e9e-8b72-028acdea9ad8_isrForCrosstalkSources_1125916_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/3123e2cb-0017-4e9e-8b72-028acdea9ad8_isrForCrosstalkSources_1125916_4.stderr parsl.dataflow.dflow DEBUG: Task 1897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1897 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1897 has memoization hash 73b017e69d49daf68969c3bbd3169519 parsl.dataflow.memoization INFO: Task 1897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1897 try 0 launched on executor multi with executor id 1898 parsl.dataflow.dflow INFO: Standard output for task 1897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/debb173a-38cc-48f3-bb47-f0e961158802_isrForCrosstalkSources_1125892_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/debb173a-38cc-48f3-bb47-f0e961158802_isrForCrosstalkSources_1125892_31.stderr parsl.dataflow.dflow DEBUG: Task 1898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1898 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1898 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1898 has memoization hash fbbb3b5808cd443f720e2de8462e4375 parsl.dataflow.memoization INFO: Task 1898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1898 try 0 launched on executor multi with executor id 1899 parsl.dataflow.dflow INFO: Standard output for task 1898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0f1ba34b-de3e-4326-889c-f32f6eb276a3_isrForCrosstalkSources_1125904_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0f1ba34b-de3e-4326-889c-f32f6eb276a3_isrForCrosstalkSources_1125904_16.stderr parsl.dataflow.dflow DEBUG: Task 1899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1899 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1899 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1899 has memoization hash f860626ad740df175e7cda9b9a250c8a parsl.dataflow.memoization INFO: Task 1899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1899 try 0 launched on executor multi with executor id 1900 parsl.dataflow.dflow INFO: Standard output for task 1899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/7fb90f8c-9fc5-49a4-abc2-27576b2ba298_isrForCrosstalkSources_1125901_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/7fb90f8c-9fc5-49a4-abc2-27576b2ba298_isrForCrosstalkSources_1125901_40.stderr parsl.dataflow.dflow DEBUG: Task 1900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1900 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1900 has memoization hash 5994e4f19b0b7e23f923f6231b0fb0f3 parsl.dataflow.memoization INFO: Task 1900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1900 try 0 launched on executor multi with executor id 1901 parsl.dataflow.dflow INFO: Standard output for task 1900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e4f92003-31b1-44f8-95d5-228a720327e0_isrForCrosstalkSources_1125901_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e4f92003-31b1-44f8-95d5-228a720327e0_isrForCrosstalkSources_1125901_41.stderr parsl.dataflow.dflow DEBUG: Task 1901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1901 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1901 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1901 has memoization hash cb71ea3c0f63b739c8b248539b6d8bd0 parsl.dataflow.memoization INFO: Task 1901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1901 try 0 launched on executor multi with executor id 1902 parsl.dataflow.dflow INFO: Standard output for task 1901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/5caa3c1d-9f92-4175-ab09-25182e5fc41b_isrForCrosstalkSources_1125897_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/5caa3c1d-9f92-4175-ab09-25182e5fc41b_isrForCrosstalkSources_1125897_34.stderr parsl.dataflow.dflow DEBUG: Task 1902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1902 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1902 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1902 has memoization hash f2de96118f2560f584a5e4c2402d6c92 parsl.dataflow.memoization INFO: Task 1902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1902 try 0 launched on executor multi with executor id 1903 parsl.dataflow.dflow INFO: Standard output for task 1902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/68c762ca-d5f3-4474-b72e-a9bc0732ac23_isrForCrosstalkSources_1125939_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/68c762ca-d5f3-4474-b72e-a9bc0732ac23_isrForCrosstalkSources_1125939_50.stderr parsl.dataflow.dflow DEBUG: Task 1903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1903 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1903 has memoization hash 50cb919832dcd8a9f95ee851ecc1ba9c parsl.dataflow.memoization INFO: Task 1903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1903 try 0 launched on executor multi with executor id 1904 parsl.dataflow.dflow INFO: Standard output for task 1903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/d50d7ff5-b57f-4c51-92b2-46ff5cd74d26_isrForCrosstalkSources_1125888_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/d50d7ff5-b57f-4c51-92b2-46ff5cd74d26_isrForCrosstalkSources_1125888_33.stderr parsl.dataflow.dflow DEBUG: Task 1904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1904 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1904 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1904 has memoization hash d875cef97c4793fac5eba3dd250162dd parsl.dataflow.memoization INFO: Task 1904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1904 try 0 launched on executor multi with executor id 1905 parsl.dataflow.dflow INFO: Standard output for task 1904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/6abcca8d-c133-44c0-aee2-18b28de189d1_isrForCrosstalkSources_1125893_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/6abcca8d-c133-44c0-aee2-18b28de189d1_isrForCrosstalkSources_1125893_9.stderr parsl.dataflow.dflow DEBUG: Task 1905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1905 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1905 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1905 has memoization hash d1d311dbe4c374ae35db8653decafd02 parsl.dataflow.memoization INFO: Task 1905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1905 try 0 launched on executor multi with executor id 1906 parsl.dataflow.dflow INFO: Standard output for task 1905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/e7e2f45c-bab8-49ae-9f30-4f0ed3b34248_isrForCrosstalkSources_1125906_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/e7e2f45c-bab8-49ae-9f30-4f0ed3b34248_isrForCrosstalkSources_1125906_40.stderr parsl.dataflow.dflow DEBUG: Task 1906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1906 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1906 has memoization hash 86723f2a60b27a0413a66d2f64a7ebdd parsl.dataflow.memoization INFO: Task 1906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1906 try 0 launched on executor multi with executor id 1907 parsl.dataflow.dflow INFO: Standard output for task 1906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/332f5514-d9c6-49c1-a027-74a2d867a1e0_isrForCrosstalkSources_1125929_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/332f5514-d9c6-49c1-a027-74a2d867a1e0_isrForCrosstalkSources_1125929_4.stderr parsl.dataflow.dflow DEBUG: Task 1907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1907 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1907 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1907 has memoization hash 404240f0149d103f544ecd9cc1a70acf parsl.dataflow.memoization INFO: Task 1907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1907 try 0 launched on executor multi with executor id 1908 parsl.dataflow.dflow INFO: Standard output for task 1907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/f5946148-c50f-4946-a709-ef5b94390588_isrForCrosstalkSources_1125889_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/f5946148-c50f-4946-a709-ef5b94390588_isrForCrosstalkSources_1125889_36.stderr parsl.dataflow.dflow DEBUG: Task 1908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1908 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1908 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1908 has memoization hash f58ae14afd5b58b29acaa91b729b9506 parsl.dataflow.memoization INFO: Task 1908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1908 try 0 launched on executor multi with executor id 1909 parsl.dataflow.dflow INFO: Standard output for task 1908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/5c65be0f-6dfa-49ad-869c-70206107a27c_isrForCrosstalkSources_1125914_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/5c65be0f-6dfa-49ad-869c-70206107a27c_isrForCrosstalkSources_1125914_26.stderr parsl.dataflow.dflow DEBUG: Task 1909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1909 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1909 has memoization hash 3d086e7e79558ac0c464cc4c54e8867c parsl.dataflow.memoization INFO: Task 1909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1909 try 0 launched on executor multi with executor id 1910 parsl.dataflow.dflow INFO: Standard output for task 1909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/9fb951ce-a124-4d53-bc06-c8839a5a453f_isrForCrosstalkSources_1125933_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/9fb951ce-a124-4d53-bc06-c8839a5a453f_isrForCrosstalkSources_1125933_11.stderr parsl.dataflow.dflow DEBUG: Task 1910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1910 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1910 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1910 has memoization hash c3d4ea20a35f53bf7dedb4880a3cb401 parsl.dataflow.memoization INFO: Task 1910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1910 try 0 launched on executor multi with executor id 1911 parsl.dataflow.dflow INFO: Standard output for task 1910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/cadf1e08-d7d5-4e37-a34d-bf4eed63de15_isrForCrosstalkSources_1125940_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/cadf1e08-d7d5-4e37-a34d-bf4eed63de15_isrForCrosstalkSources_1125940_31.stderr parsl.dataflow.dflow DEBUG: Task 1911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1911 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1911 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1911 has memoization hash e248d19e97879a80052f1b633525ff9d parsl.dataflow.memoization INFO: Task 1911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1911 try 0 launched on executor multi with executor id 1912 parsl.dataflow.dflow INFO: Standard output for task 1911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/0abaecf6-4180-4c00-89e4-c878bd385000_isrForCrosstalkSources_1125912_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/0abaecf6-4180-4c00-89e4-c878bd385000_isrForCrosstalkSources_1125912_62.stderr parsl.dataflow.dflow DEBUG: Task 1912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1912 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1912 has memoization hash 3e62724708b1c8e252bb85cb087d136b parsl.dataflow.memoization INFO: Task 1912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1912 try 0 launched on executor multi with executor id 1913 parsl.dataflow.dflow INFO: Standard output for task 1912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/f746dcaa-59ac-444f-882f-af8ea6a2ed4d_isrForCrosstalkSources_1125934_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/f746dcaa-59ac-444f-882f-af8ea6a2ed4d_isrForCrosstalkSources_1125934_46.stderr parsl.dataflow.dflow DEBUG: Task 1913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1913 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1913 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1913 has memoization hash 53e90528c0450b050cf08ebeab875171 parsl.dataflow.memoization INFO: Task 1913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1913 try 0 launched on executor multi with executor id 1914 parsl.dataflow.dflow INFO: Standard output for task 1913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/596d0218-9c50-4937-9c03-630939588634_isrForCrosstalkSources_1125913_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/596d0218-9c50-4937-9c03-630939588634_isrForCrosstalkSources_1125913_10.stderr parsl.dataflow.dflow DEBUG: Task 1914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1914 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1914 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1914 has memoization hash 8855337801fea3e64f8dcce79ab7854e parsl.dataflow.memoization INFO: Task 1914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1914 try 0 launched on executor multi with executor id 1915 parsl.dataflow.dflow INFO: Standard output for task 1914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/182e3257-a94c-47d6-bd35-5bf1e048314f_isrForCrosstalkSources_1125937_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/182e3257-a94c-47d6-bd35-5bf1e048314f_isrForCrosstalkSources_1125937_38.stderr parsl.dataflow.dflow DEBUG: Task 1915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1915 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1915 has memoization hash 775e68e2cd94a456a485a282a0308cf9 parsl.dataflow.memoization INFO: Task 1915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1915 try 0 launched on executor multi with executor id 1916 parsl.dataflow.dflow INFO: Standard output for task 1915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e406a09e-0d7e-464f-8a84-914499a73d55_isrForCrosstalkSources_1125942_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e406a09e-0d7e-464f-8a84-914499a73d55_isrForCrosstalkSources_1125942_17.stderr parsl.dataflow.dflow DEBUG: Task 1916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1916 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1916 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1916 has memoization hash 6aae1ea6fd87f9964ce00f59bc5533ee parsl.dataflow.memoization INFO: Task 1916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1916 try 0 launched on executor multi with executor id 1917 parsl.dataflow.dflow INFO: Standard output for task 1916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/3496d3c5-9663-4eb5-94c3-f8dc9e19ba53_isrForCrosstalkSources_1125890_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/3496d3c5-9663-4eb5-94c3-f8dc9e19ba53_isrForCrosstalkSources_1125890_40.stderr parsl.dataflow.dflow DEBUG: Task 1917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1917 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1917 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1917 has memoization hash f95140a27526d4c1f2967acbc008f612 parsl.dataflow.memoization INFO: Task 1917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1917 try 0 launched on executor multi with executor id 1918 parsl.dataflow.dflow INFO: Standard output for task 1917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/eb0d66b7-6b5f-444e-9754-0f03941b2d63_isrForCrosstalkSources_1125943_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/eb0d66b7-6b5f-444e-9754-0f03941b2d63_isrForCrosstalkSources_1125943_23.stderr parsl.dataflow.dflow DEBUG: Task 1918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1918 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1918 has memoization hash d148c3281b1fe3b64736fb9262f11b63 parsl.dataflow.memoization INFO: Task 1918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1918 try 0 launched on executor multi with executor id 1919 parsl.dataflow.dflow INFO: Standard output for task 1918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/92a09669-7d38-40fb-ade1-7a88c0116812_isrForCrosstalkSources_1125938_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/92a09669-7d38-40fb-ade1-7a88c0116812_isrForCrosstalkSources_1125938_59.stderr parsl.dataflow.dflow DEBUG: Task 1919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1919 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1919 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1919 has memoization hash ca7b83770a98ea6044f7e41cd37574f2 parsl.dataflow.memoization INFO: Task 1919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1919 try 0 launched on executor multi with executor id 1920 parsl.dataflow.dflow INFO: Standard output for task 1919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/56048bf6-8870-45bf-948f-e1e636100206_isrForCrosstalkSources_1125930_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/56048bf6-8870-45bf-948f-e1e636100206_isrForCrosstalkSources_1125930_37.stderr parsl.dataflow.dflow DEBUG: Task 1920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1920 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1920 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1920 has memoization hash 06d343609ce133aad0ef2788c7f068d5 parsl.dataflow.memoization INFO: Task 1920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1920 try 0 launched on executor multi with executor id 1921 parsl.dataflow.dflow INFO: Standard output for task 1920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/9d2589c6-b4f5-45dd-b695-43c407cac107_isrForCrosstalkSources_1125919_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/9d2589c6-b4f5-45dd-b695-43c407cac107_isrForCrosstalkSources_1125919_30.stderr parsl.dataflow.dflow DEBUG: Task 1921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1921 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1921 has memoization hash 669552bd460a642a345f80c311f9ff51 parsl.dataflow.memoization INFO: Task 1921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1921 try 0 launched on executor multi with executor id 1922 parsl.dataflow.dflow INFO: Standard output for task 1921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/0da4367c-a84f-4e6f-86a9-79f31954bc9c_isrForCrosstalkSources_1125923_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/0da4367c-a84f-4e6f-86a9-79f31954bc9c_isrForCrosstalkSources_1125923_50.stderr parsl.dataflow.dflow DEBUG: Task 1922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1922 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1922 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1922 has memoization hash 6b793bcfae9e00f8a0e3ee9bcfc5e0ca parsl.dataflow.memoization INFO: Task 1922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1922 try 0 launched on executor multi with executor id 1923 parsl.dataflow.dflow INFO: Standard output for task 1922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ea5cce1a-080d-463e-ba90-e556b379fdeb_isrForCrosstalkSources_1125939_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ea5cce1a-080d-463e-ba90-e556b379fdeb_isrForCrosstalkSources_1125939_55.stderr parsl.dataflow.dflow DEBUG: Task 1923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1923 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1923 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1923 has memoization hash 0fc732709efab9961fb9889baa7fa363 parsl.dataflow.memoization INFO: Task 1923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1923 try 0 launched on executor multi with executor id 1924 parsl.dataflow.dflow INFO: Standard output for task 1923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/65107593-d678-4203-9793-a9f1e0331fe8_isrForCrosstalkSources_1125945_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/65107593-d678-4203-9793-a9f1e0331fe8_isrForCrosstalkSources_1125945_1.stderr parsl.dataflow.dflow DEBUG: Task 1924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1924 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1924 has memoization hash d00d4f129c9a007d2069cf39eca84cd8 parsl.dataflow.memoization INFO: Task 1924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1924 try 0 launched on executor multi with executor id 1925 parsl.dataflow.dflow INFO: Standard output for task 1924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/08b2fa34-409a-4083-8bba-6ce6a8a14c0c_isrForCrosstalkSources_1125894_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/08b2fa34-409a-4083-8bba-6ce6a8a14c0c_isrForCrosstalkSources_1125894_22.stderr parsl.dataflow.dflow DEBUG: Task 1925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1925 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1925 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1925 has memoization hash 3ae65b5757f069e83d980c9ad6554e67 parsl.dataflow.memoization INFO: Task 1925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1925 try 0 launched on executor multi with executor id 1926 parsl.dataflow.dflow INFO: Standard output for task 1925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/c9b99e80-0bdd-4ff6-a3c0-d7be8b4f5047_isrForCrosstalkSources_1125912_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/c9b99e80-0bdd-4ff6-a3c0-d7be8b4f5047_isrForCrosstalkSources_1125912_7.stderr parsl.dataflow.dflow DEBUG: Task 1926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1926 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1926 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1926 has memoization hash cff172596c595998e747b45b7315c80e parsl.dataflow.memoization INFO: Task 1926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1926 try 0 launched on executor multi with executor id 1927 parsl.dataflow.dflow INFO: Standard output for task 1926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/5a3b338f-20fc-4393-82fc-6acb18d1f37e_isrForCrosstalkSources_1125939_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/5a3b338f-20fc-4393-82fc-6acb18d1f37e_isrForCrosstalkSources_1125939_47.stderr parsl.dataflow.dflow DEBUG: Task 1927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1927 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1927 has memoization hash a51c052e624406e4748591fc6a9d9bfe parsl.dataflow.memoization INFO: Task 1927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1927 try 0 launched on executor multi with executor id 1928 parsl.dataflow.dflow INFO: Standard output for task 1927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/f337b54d-a436-4d1c-9a51-a0e529580713_isrForCrosstalkSources_1125932_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/f337b54d-a436-4d1c-9a51-a0e529580713_isrForCrosstalkSources_1125932_9.stderr parsl.dataflow.dflow DEBUG: Task 1928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1928 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1928 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1928 has memoization hash e2b27d9bda7b102d2b809bc5780f2dd0 parsl.dataflow.memoization INFO: Task 1928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1928 try 0 launched on executor multi with executor id 1929 parsl.dataflow.dflow INFO: Standard output for task 1928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/2cba0780-ccab-43b3-a118-34621ee2328f_isrForCrosstalkSources_1125928_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/2cba0780-ccab-43b3-a118-34621ee2328f_isrForCrosstalkSources_1125928_51.stderr parsl.dataflow.dflow DEBUG: Task 1929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1929 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1929 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1929 has memoization hash 535c49ef46926abc40615980527a56df parsl.dataflow.memoization INFO: Task 1929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1929 try 0 launched on executor multi with executor id 1930 parsl.dataflow.dflow INFO: Standard output for task 1929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/2c81bcc6-c907-4b99-a4c3-47ddadfb5360_isrForCrosstalkSources_1125903_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/2c81bcc6-c907-4b99-a4c3-47ddadfb5360_isrForCrosstalkSources_1125903_26.stderr parsl.dataflow.dflow DEBUG: Task 1930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1930 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1930 has memoization hash 3ba742a64f7a74fedd80a0d029a4d966 parsl.dataflow.memoization INFO: Task 1930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1930 try 0 launched on executor multi with executor id 1931 parsl.dataflow.dflow INFO: Standard output for task 1930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/873dffcc-227c-443f-ae8c-8d3a9c940e1e_isrForCrosstalkSources_1125897_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/873dffcc-227c-443f-ae8c-8d3a9c940e1e_isrForCrosstalkSources_1125897_23.stderr parsl.dataflow.dflow DEBUG: Task 1931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1931 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1931 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1931 has memoization hash 7b53decaafbaa43f664d6d3b916c873a parsl.dataflow.memoization INFO: Task 1931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1931 try 0 launched on executor multi with executor id 1932 parsl.dataflow.dflow INFO: Standard output for task 1931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/ea8adfe4-34d8-4348-a351-8eb11c6395be_isrForCrosstalkSources_1125909_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/ea8adfe4-34d8-4348-a351-8eb11c6395be_isrForCrosstalkSources_1125909_30.stderr parsl.dataflow.dflow DEBUG: Task 1932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1932 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1932 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1932 has memoization hash f3a82a2cd51ab8ad98563ae94e2fd841 parsl.dataflow.memoization INFO: Task 1932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1932 try 0 launched on executor multi with executor id 1933 parsl.dataflow.dflow INFO: Standard output for task 1932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/70c9658e-c61c-4f91-b5a7-972290625396_isrForCrosstalkSources_1125938_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/70c9658e-c61c-4f91-b5a7-972290625396_isrForCrosstalkSources_1125938_26.stderr parsl.dataflow.dflow DEBUG: Task 1933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1933 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1933 has memoization hash 577978f2284501dcea8f4710be3d451a parsl.dataflow.memoization INFO: Task 1933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1933 try 0 launched on executor multi with executor id 1934 parsl.dataflow.dflow INFO: Standard output for task 1933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/5fcabf08-6be7-465c-a3f2-cff56a99bd43_isrForCrosstalkSources_1125901_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/5fcabf08-6be7-465c-a3f2-cff56a99bd43_isrForCrosstalkSources_1125901_23.stderr parsl.dataflow.dflow DEBUG: Task 1934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1934 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1934 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1934 has memoization hash 0ff350cb67b9328706d6ecc5d845df2b parsl.dataflow.memoization INFO: Task 1934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1934 try 0 launched on executor multi with executor id 1935 parsl.dataflow.dflow INFO: Standard output for task 1934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/f27dd882-c714-42f9-90e7-b0dd8f243834_isrForCrosstalkSources_1125902_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/f27dd882-c714-42f9-90e7-b0dd8f243834_isrForCrosstalkSources_1125902_15.stderr parsl.dataflow.dflow DEBUG: Task 1935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1935 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1935 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1935 has memoization hash ccfe81a51b3419d1a32608ab988cd4d1 parsl.dataflow.memoization INFO: Task 1935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1935 try 0 launched on executor multi with executor id 1936 parsl.dataflow.dflow INFO: Standard output for task 1935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/d33bf2c8-e880-4eb2-ad51-f6ce024d7a78_isrForCrosstalkSources_1125937_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/d33bf2c8-e880-4eb2-ad51-f6ce024d7a78_isrForCrosstalkSources_1125937_37.stderr parsl.dataflow.dflow DEBUG: Task 1936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1936 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1936 has memoization hash 8296ea10b1870bb227514ad22aa93507 parsl.dataflow.memoization INFO: Task 1936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1936 try 0 launched on executor multi with executor id 1937 parsl.dataflow.dflow INFO: Standard output for task 1936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/b1d39a75-8fd5-4527-a512-222dab1046da_isrForCrosstalkSources_1125915_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/b1d39a75-8fd5-4527-a512-222dab1046da_isrForCrosstalkSources_1125915_49.stderr parsl.dataflow.dflow DEBUG: Task 1937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1937 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1937 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1937 has memoization hash 47897892e917ef1076e8320e3ce74dfa parsl.dataflow.memoization INFO: Task 1937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1937 try 0 launched on executor multi with executor id 1938 parsl.dataflow.dflow INFO: Standard output for task 1937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/55924b7a-540a-4aac-b0a2-51317a5d1aca_isrForCrosstalkSources_1125944_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/55924b7a-540a-4aac-b0a2-51317a5d1aca_isrForCrosstalkSources_1125944_40.stderr parsl.dataflow.dflow DEBUG: Task 1938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1938 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1938 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1938 has memoization hash 89a83d084141eca521c9e8472d63c086 parsl.dataflow.memoization INFO: Task 1938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1938 try 0 launched on executor multi with executor id 1939 parsl.dataflow.dflow INFO: Standard output for task 1938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/9bb03b0b-38b9-41d6-9426-2d8052590169_isrForCrosstalkSources_1125922_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/9bb03b0b-38b9-41d6-9426-2d8052590169_isrForCrosstalkSources_1125922_15.stderr parsl.dataflow.dflow DEBUG: Task 1939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1939 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1939 has memoization hash f69ea10ec1e8badfebff0052315d2ce5 parsl.dataflow.memoization INFO: Task 1939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1939 try 0 launched on executor multi with executor id 1940 parsl.dataflow.dflow INFO: Standard output for task 1939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/82e0710c-1d43-491d-b32e-240db8460736_isrForCrosstalkSources_1125920_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/82e0710c-1d43-491d-b32e-240db8460736_isrForCrosstalkSources_1125920_23.stderr parsl.dataflow.dflow DEBUG: Task 1940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1940 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1940 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1940 has memoization hash 828fc0efde46f380d576e821cfced6f8 parsl.dataflow.memoization INFO: Task 1940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1940 try 0 launched on executor multi with executor id 1941 parsl.dataflow.dflow INFO: Standard output for task 1940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e85eff13-72cb-4d14-bc03-5a4825f94770_isrForCrosstalkSources_1125916_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e85eff13-72cb-4d14-bc03-5a4825f94770_isrForCrosstalkSources_1125916_3.stderr parsl.dataflow.dflow DEBUG: Task 1941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1941 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1941 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1941 has memoization hash c96475dcb22255dcff97091353ba6269 parsl.dataflow.memoization INFO: Task 1941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1941 try 0 launched on executor multi with executor id 1942 parsl.dataflow.dflow INFO: Standard output for task 1941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/61b5e19c-79a1-44b5-8740-67db88170a8f_isrForCrosstalkSources_1125926_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/61b5e19c-79a1-44b5-8740-67db88170a8f_isrForCrosstalkSources_1125926_60.stderr parsl.dataflow.dflow DEBUG: Task 1942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1942 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1942 has memoization hash d1768b156cee3afe59e2d1612dde9502 parsl.dataflow.memoization INFO: Task 1942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1942 try 0 launched on executor multi with executor id 1943 parsl.dataflow.dflow INFO: Standard output for task 1942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/0f0d6bae-6fc5-459a-809d-3bc6e465e37e_isrForCrosstalkSources_1125888_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/0f0d6bae-6fc5-459a-809d-3bc6e465e37e_isrForCrosstalkSources_1125888_39.stderr parsl.dataflow.dflow DEBUG: Task 1943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1943 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1943 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1943 has memoization hash fd804f04252b8315cbbaad123074cfdf parsl.dataflow.memoization INFO: Task 1943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3994ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1943 try 0 launched on executor multi with executor id 1944 parsl.dataflow.dflow INFO: Standard output for task 1943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/942680e4-ae2d-448c-9f3b-b0315b17ebcb_isrForCrosstalkSources_1125889_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/942680e4-ae2d-448c-9f3b-b0315b17ebcb_isrForCrosstalkSources_1125889_11.stderr parsl.dataflow.dflow DEBUG: Task 1944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1944 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1944 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1944 has memoization hash 86e612321bf8eb3fd6279134b7549412 parsl.dataflow.memoization INFO: Task 1944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1944 try 0 launched on executor multi with executor id 1945 parsl.dataflow.dflow INFO: Standard output for task 1944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d4056072-718e-4c84-bbd4-23ad9d3fabfd_isrForCrosstalkSources_1125905_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d4056072-718e-4c84-bbd4-23ad9d3fabfd_isrForCrosstalkSources_1125905_35.stderr parsl.dataflow.dflow DEBUG: Task 1945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1945 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1945 has memoization hash 69bad61032e477c42078b3ed2180b959 parsl.dataflow.memoization INFO: Task 1945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1945 try 0 launched on executor multi with executor id 1946 parsl.dataflow.dflow INFO: Standard output for task 1945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/cd6d3c23-b37c-45f7-9b0e-753938d5006b_isrForCrosstalkSources_1125914_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/cd6d3c23-b37c-45f7-9b0e-753938d5006b_isrForCrosstalkSources_1125914_10.stderr parsl.dataflow.dflow DEBUG: Task 1946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1946 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1946 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1946 has memoization hash 65c45992e83d6b33bc0ffe090f4f1340 parsl.dataflow.memoization INFO: Task 1946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1946 try 0 launched on executor multi with executor id 1947 parsl.dataflow.dflow INFO: Standard output for task 1946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/1d61ec42-4d0b-49b9-8683-884e3a59e288_isrForCrosstalkSources_1125914_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/1d61ec42-4d0b-49b9-8683-884e3a59e288_isrForCrosstalkSources_1125914_47.stderr parsl.dataflow.dflow DEBUG: Task 1947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1947 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1947 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1947 has memoization hash 0ede1170f34561d18a496936bf43d746 parsl.dataflow.memoization INFO: Task 1947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1947 try 0 launched on executor multi with executor id 1948 parsl.dataflow.dflow INFO: Standard output for task 1947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/b02eeb17-4423-49a2-b07a-39fd62b3a31c_isrForCrosstalkSources_1125914_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/b02eeb17-4423-49a2-b07a-39fd62b3a31c_isrForCrosstalkSources_1125914_23.stderr parsl.dataflow.dflow DEBUG: Task 1948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1948 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1948 has memoization hash 01996b48a561cb1dfd17f5e8838e0b6e parsl.dataflow.memoization INFO: Task 1948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1948 try 0 launched on executor multi with executor id 1949 parsl.dataflow.dflow INFO: Standard output for task 1948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3f00b221-fb60-4f2b-a826-99284f7996f1_isrForCrosstalkSources_1125912_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3f00b221-fb60-4f2b-a826-99284f7996f1_isrForCrosstalkSources_1125912_56.stderr parsl.dataflow.dflow DEBUG: Task 1949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1949 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1949 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1949 has memoization hash fc5f1b48c09cbdc2661de86dd6a330e2 parsl.dataflow.memoization INFO: Task 1949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1949 try 0 launched on executor multi with executor id 1950 parsl.dataflow.dflow INFO: Standard output for task 1949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/c53d8bcc-ee5e-4b37-8a56-27029d9d1673_isrForCrosstalkSources_1125927_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/c53d8bcc-ee5e-4b37-8a56-27029d9d1673_isrForCrosstalkSources_1125927_27.stderr parsl.dataflow.dflow DEBUG: Task 1950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1950 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1950 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1950 has memoization hash 534d41fe24a03cecd865390ad1ee9a28 parsl.dataflow.memoization INFO: Task 1950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1950 try 0 launched on executor multi with executor id 1951 parsl.dataflow.dflow INFO: Standard output for task 1950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/ca386bd1-12f8-4f5a-865c-f7a51318f55b_isrForCrosstalkSources_1125925_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/ca386bd1-12f8-4f5a-865c-f7a51318f55b_isrForCrosstalkSources_1125925_14.stderr parsl.dataflow.dflow DEBUG: Task 1951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1951 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1951 has memoization hash b1a37add4119624aec5970a838a152bd parsl.dataflow.memoization INFO: Task 1951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1951 try 0 launched on executor multi with executor id 1952 parsl.dataflow.dflow INFO: Standard output for task 1951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/9f93ffb0-bfdd-48b4-8748-77d91b7e8a94_isrForCrosstalkSources_1125932_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/9f93ffb0-bfdd-48b4-8748-77d91b7e8a94_isrForCrosstalkSources_1125932_57.stderr parsl.dataflow.dflow DEBUG: Task 1952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1952 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1952 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1952 has memoization hash 63fe7cb42166d7f6756ffb9702d9e9ee parsl.dataflow.memoization INFO: Task 1952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1952 try 0 launched on executor multi with executor id 1953 parsl.dataflow.dflow INFO: Standard output for task 1952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/9ab75946-a69d-4efa-9187-3161324cf867_isrForCrosstalkSources_1125913_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/9ab75946-a69d-4efa-9187-3161324cf867_isrForCrosstalkSources_1125913_58.stderr parsl.dataflow.dflow DEBUG: Task 1953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1953 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1953 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1953 has memoization hash 090f7948636cf2232f901857ced405e3 parsl.dataflow.memoization INFO: Task 1953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1953 try 0 launched on executor multi with executor id 1954 parsl.dataflow.dflow INFO: Standard output for task 1953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ca36cdeb-7e0d-4611-8733-b88c965c7fa6_isrForCrosstalkSources_1125902_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ca36cdeb-7e0d-4611-8733-b88c965c7fa6_isrForCrosstalkSources_1125902_48.stderr parsl.dataflow.dflow DEBUG: Task 1954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1954 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1954 has memoization hash 26c719b708415a03ae29fe02fbd163b4 parsl.dataflow.memoization INFO: Task 1954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1954 try 0 launched on executor multi with executor id 1955 parsl.dataflow.dflow INFO: Standard output for task 1954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/297dbfd2-fdf4-4ae2-854d-8645cd02594c_isrForCrosstalkSources_1125947_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/297dbfd2-fdf4-4ae2-854d-8645cd02594c_isrForCrosstalkSources_1125947_12.stderr parsl.dataflow.dflow DEBUG: Task 1955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1955 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1955 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1955 has memoization hash dd4ef0c7a265373b083e944ac6c3a9ad parsl.dataflow.memoization INFO: Task 1955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1955 try 0 launched on executor multi with executor id 1956 parsl.dataflow.dflow INFO: Standard output for task 1955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/a22a64f1-868e-45f0-90e2-2dce667ca0bd_isrForCrosstalkSources_1125894_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/a22a64f1-868e-45f0-90e2-2dce667ca0bd_isrForCrosstalkSources_1125894_3.stderr parsl.dataflow.dflow DEBUG: Task 1956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1956 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1956 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1956 has memoization hash a196e9accb3a16a0216bf1d849be98e9 parsl.dataflow.memoization INFO: Task 1956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1956 try 0 launched on executor multi with executor id 1957 parsl.dataflow.dflow INFO: Standard output for task 1956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/034cb056-c9ef-480a-a877-7910e7e69016_isrForCrosstalkSources_1125934_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/034cb056-c9ef-480a-a877-7910e7e69016_isrForCrosstalkSources_1125934_11.stderr parsl.dataflow.dflow DEBUG: Task 1957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1957 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1957 has memoization hash c7cd9126c20cac1383b3763d12b5d889 parsl.dataflow.memoization INFO: Task 1957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1957 try 0 launched on executor multi with executor id 1958 parsl.dataflow.dflow INFO: Standard output for task 1957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/b0db1476-9cd7-45d8-92bc-84b4ef50217d_isrForCrosstalkSources_1125917_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/b0db1476-9cd7-45d8-92bc-84b4ef50217d_isrForCrosstalkSources_1125917_49.stderr parsl.dataflow.dflow DEBUG: Task 1958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1958 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1958 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1958 has memoization hash 1a7d5a18574aa7f60e21b1b5e80ee780 parsl.dataflow.memoization INFO: Task 1958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1958 try 0 launched on executor multi with executor id 1959 parsl.dataflow.dflow INFO: Standard output for task 1958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/565bb661-b147-49c4-ad95-4529e3058231_isrForCrosstalkSources_1125943_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/565bb661-b147-49c4-ad95-4529e3058231_isrForCrosstalkSources_1125943_38.stderr parsl.dataflow.dflow DEBUG: Task 1959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1959 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1959 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1959 has memoization hash 619d5c4fdfa56fa0efb646cafa8ecdf1 parsl.dataflow.memoization INFO: Task 1959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1959 try 0 launched on executor multi with executor id 1960 parsl.dataflow.dflow INFO: Standard output for task 1959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/60c6de74-1f71-4765-b4a0-f796abe5752a_isrForCrosstalkSources_1125895_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/60c6de74-1f71-4765-b4a0-f796abe5752a_isrForCrosstalkSources_1125895_23.stderr parsl.dataflow.dflow DEBUG: Task 1960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1960 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1960 has memoization hash 14179afabdd9c9d3fe5738b93a023238 parsl.dataflow.memoization INFO: Task 1960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1960 try 0 launched on executor multi with executor id 1961 parsl.dataflow.dflow INFO: Standard output for task 1960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/321173b2-d1fc-45b6-9a06-1ea86d2002af_isrForCrosstalkSources_1125946_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/321173b2-d1fc-45b6-9a06-1ea86d2002af_isrForCrosstalkSources_1125946_35.stderr parsl.dataflow.dflow DEBUG: Task 1961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1961 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1961 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1961 has memoization hash 0b64a26ae2f85efaebb708fb4cdb3104 parsl.dataflow.memoization INFO: Task 1961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1961 try 0 launched on executor multi with executor id 1962 parsl.dataflow.dflow INFO: Standard output for task 1961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/812ba751-cf53-4de4-9877-89f73b33612c_isrForCrosstalkSources_1125930_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/812ba751-cf53-4de4-9877-89f73b33612c_isrForCrosstalkSources_1125930_43.stderr parsl.dataflow.dflow DEBUG: Task 1962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1962 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1962 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1962 has memoization hash 783f38efee29c71fb167be4a96fd731f parsl.dataflow.memoization INFO: Task 1962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1962 try 0 launched on executor multi with executor id 1963 parsl.dataflow.dflow INFO: Standard output for task 1962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/3ad071c6-4b2e-4ba7-ad80-cc8249457326_isrForCrosstalkSources_1125903_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/3ad071c6-4b2e-4ba7-ad80-cc8249457326_isrForCrosstalkSources_1125903_16.stderr parsl.dataflow.dflow DEBUG: Task 1963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1963 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1963 has memoization hash 82d0762b97116f309de02e3abbfc5509 parsl.dataflow.memoization INFO: Task 1963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1963 try 0 launched on executor multi with executor id 1964 parsl.dataflow.dflow INFO: Standard output for task 1963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/4505a721-b8b2-471d-a4e1-aea3017f5773_isrForCrosstalkSources_1125915_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/4505a721-b8b2-471d-a4e1-aea3017f5773_isrForCrosstalkSources_1125915_29.stderr parsl.dataflow.dflow DEBUG: Task 1964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1964 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1964 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1964 has memoization hash 53606fca6c5c78a1e33d3e408066a354 parsl.dataflow.memoization INFO: Task 1964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1964 try 0 launched on executor multi with executor id 1965 parsl.dataflow.dflow INFO: Standard output for task 1964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/7334268e-baae-439d-aa22-148494159373_isrForCrosstalkSources_1125930_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/7334268e-baae-439d-aa22-148494159373_isrForCrosstalkSources_1125930_56.stderr parsl.dataflow.dflow DEBUG: Task 1965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1965 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1965 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1965 has memoization hash 41e0622cbb355b727ba236d0b7598c80 parsl.dataflow.memoization INFO: Task 1965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1965 try 0 launched on executor multi with executor id 1966 parsl.dataflow.dflow INFO: Standard output for task 1965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/c84baf7e-8e03-46ee-99c0-cd0c966318df_isrForCrosstalkSources_1125908_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/c84baf7e-8e03-46ee-99c0-cd0c966318df_isrForCrosstalkSources_1125908_44.stderr parsl.dataflow.dflow DEBUG: Task 1966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1966 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1966 has memoization hash 2d94ba56acd562dfa4b13cccd9495d86 parsl.dataflow.memoization INFO: Task 1966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1966 try 0 launched on executor multi with executor id 1967 parsl.dataflow.dflow INFO: Standard output for task 1966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/f6281314-5726-4d82-9c3d-4c137269a434_isrForCrosstalkSources_1125947_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/f6281314-5726-4d82-9c3d-4c137269a434_isrForCrosstalkSources_1125947_27.stderr parsl.dataflow.dflow DEBUG: Task 1967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1967 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1967 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1967 has memoization hash 5df2215cc2b0c1c39a1c13ae0467b687 parsl.dataflow.memoization INFO: Task 1967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1967 try 0 launched on executor multi with executor id 1968 parsl.dataflow.dflow INFO: Standard output for task 1967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/2bb270cb-d78d-4e44-94b5-53126a3cfe58_isrForCrosstalkSources_1125946_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/2bb270cb-d78d-4e44-94b5-53126a3cfe58_isrForCrosstalkSources_1125946_14.stderr parsl.dataflow.dflow DEBUG: Task 1968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1968 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1968 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1968 has memoization hash a82cbb261c0de09143f03dd8cd285bf9 parsl.dataflow.memoization INFO: Task 1968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1968 try 0 launched on executor multi with executor id 1969 parsl.dataflow.dflow INFO: Standard output for task 1968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/0dec1cde-46a7-45b8-9f99-059a576462c0_isrForCrosstalkSources_1125893_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/0dec1cde-46a7-45b8-9f99-059a576462c0_isrForCrosstalkSources_1125893_62.stderr parsl.dataflow.dflow DEBUG: Task 1969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1969 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1969 has memoization hash bcae32604eb00644edd9a2be4c078e4c parsl.dataflow.memoization INFO: Task 1969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1969 try 0 launched on executor multi with executor id 1970 parsl.dataflow.dflow INFO: Standard output for task 1969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/7aaf0a1c-605b-4c1c-95b6-89ed69e8af83_isrForCrosstalkSources_1125913_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/7aaf0a1c-605b-4c1c-95b6-89ed69e8af83_isrForCrosstalkSources_1125913_44.stderr parsl.dataflow.dflow DEBUG: Task 1970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1970 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1970 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1970 has memoization hash 635b4feb7ad8ee1256bb422cd03f3426 parsl.dataflow.memoization INFO: Task 1970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1970 try 0 launched on executor multi with executor id 1971 parsl.dataflow.dflow INFO: Standard output for task 1970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/eaa8fdf6-58f3-4822-bae0-01e446110f36_isrForCrosstalkSources_1125930_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/eaa8fdf6-58f3-4822-bae0-01e446110f36_isrForCrosstalkSources_1125930_55.stderr parsl.dataflow.dflow DEBUG: Task 1971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1971 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1971 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1971 has memoization hash a5152f5ba6301904a189b4ecdfd5a128 parsl.dataflow.memoization INFO: Task 1971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1971 try 0 launched on executor multi with executor id 1972 parsl.dataflow.dflow INFO: Standard output for task 1971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/06c9a8a4-6171-478d-8759-9be34671b41c_isrForCrosstalkSources_1125919_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/06c9a8a4-6171-478d-8759-9be34671b41c_isrForCrosstalkSources_1125919_9.stderr parsl.dataflow.dflow DEBUG: Task 1972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1972 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1972 has memoization hash c6095c96cf3a6a99aaf68eb2b8737c4f parsl.dataflow.memoization INFO: Task 1972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1972 try 0 launched on executor multi with executor id 1973 parsl.dataflow.dflow INFO: Standard output for task 1972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/543498de-133f-4395-b138-a8f1f606b3ca_isrForCrosstalkSources_1125909_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/543498de-133f-4395-b138-a8f1f606b3ca_isrForCrosstalkSources_1125909_52.stderr parsl.dataflow.dflow DEBUG: Task 1973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1973 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1973 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1973 has memoization hash 4869cddae430017367ee5708c3da295b parsl.dataflow.memoization INFO: Task 1973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1973 try 0 launched on executor multi with executor id 1974 parsl.dataflow.dflow INFO: Standard output for task 1973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/10c26f11-161f-4a3e-a013-538b8bb7de93_isrForCrosstalkSources_1125940_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/10c26f11-161f-4a3e-a013-538b8bb7de93_isrForCrosstalkSources_1125940_6.stderr parsl.dataflow.dflow DEBUG: Task 1974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1974 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1974 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1974 has memoization hash 52065301d3b86094a3fa51489fb6f7b9 parsl.dataflow.memoization INFO: Task 1974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1974 try 0 launched on executor multi with executor id 1975 parsl.dataflow.dflow INFO: Standard output for task 1974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/e48156e1-1440-41be-85cb-f6d056ed6259_isrForCrosstalkSources_1125935_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/e48156e1-1440-41be-85cb-f6d056ed6259_isrForCrosstalkSources_1125935_61.stderr parsl.dataflow.dflow DEBUG: Task 1975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1975 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1975 has memoization hash f120c42f21f2ed5548835ac789cf5c51 parsl.dataflow.memoization INFO: Task 1975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1975 try 0 launched on executor multi with executor id 1976 parsl.dataflow.dflow INFO: Standard output for task 1975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/156c9d77-c360-4ac9-84ce-6830e88b7a40_isrForCrosstalkSources_1125939_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/156c9d77-c360-4ac9-84ce-6830e88b7a40_isrForCrosstalkSources_1125939_11.stderr parsl.dataflow.dflow DEBUG: Task 1976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1976 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1976 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1976 has memoization hash 43896c27e2f5637c44cadaff9192acac parsl.dataflow.memoization INFO: Task 1976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1976 try 0 launched on executor multi with executor id 1977 parsl.dataflow.dflow INFO: Standard output for task 1976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/eccecce3-b3d0-4ad2-ab45-64b4447b19c2_isrForCrosstalkSources_1125926_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/eccecce3-b3d0-4ad2-ab45-64b4447b19c2_isrForCrosstalkSources_1125926_44.stderr parsl.dataflow.dflow DEBUG: Task 1977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1977 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1977 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1977 has memoization hash 39c9960fddddcdccdab668075e1fc7a8 parsl.dataflow.memoization INFO: Task 1977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1977 try 0 launched on executor multi with executor id 1978 parsl.dataflow.dflow INFO: Standard output for task 1977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/310f073a-ea81-4f4c-8f52-53cbcef5b626_isrForCrosstalkSources_1125945_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/310f073a-ea81-4f4c-8f52-53cbcef5b626_isrForCrosstalkSources_1125945_54.stderr parsl.dataflow.dflow DEBUG: Task 1978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1978 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1978 has memoization hash dbb41a7d7939ad793e20cf41403e4c73 parsl.dataflow.memoization INFO: Task 1978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1978 try 0 launched on executor multi with executor id 1979 parsl.dataflow.dflow INFO: Standard output for task 1978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/4b5d93dc-6959-40aa-bc93-34fc24478b2f_isrForCrosstalkSources_1125896_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/4b5d93dc-6959-40aa-bc93-34fc24478b2f_isrForCrosstalkSources_1125896_37.stderr parsl.dataflow.dflow DEBUG: Task 1979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1979 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1979 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1979 has memoization hash d1d7e4c8a5429b87eec132d247d34251 parsl.dataflow.memoization INFO: Task 1979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1979 try 0 launched on executor multi with executor id 1980 parsl.dataflow.dflow INFO: Standard output for task 1979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/d9f8399f-7c48-49d1-a143-aa74a80681d6_isrForCrosstalkSources_1125903_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/d9f8399f-7c48-49d1-a143-aa74a80681d6_isrForCrosstalkSources_1125903_1.stderr parsl.dataflow.dflow DEBUG: Task 1980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1980 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1980 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1980 has memoization hash 11f18a32f501c3579ef126d6135f011f parsl.dataflow.memoization INFO: Task 1980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1980 try 0 launched on executor multi with executor id 1981 parsl.dataflow.dflow INFO: Standard output for task 1980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/eeb0f489-a34c-483f-9bcf-696e4f7e6e66_isrForCrosstalkSources_1125890_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/eeb0f489-a34c-483f-9bcf-696e4f7e6e66_isrForCrosstalkSources_1125890_36.stderr parsl.dataflow.dflow DEBUG: Task 1981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1981 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1981 has memoization hash ad3ac02c4efed6b38322061025e706cf parsl.dataflow.memoization INFO: Task 1981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1981 try 0 launched on executor multi with executor id 1982 parsl.dataflow.dflow INFO: Standard output for task 1981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/adc1474f-55e2-4a65-ab5d-02cecf253d47_isrForCrosstalkSources_1125936_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/adc1474f-55e2-4a65-ab5d-02cecf253d47_isrForCrosstalkSources_1125936_34.stderr parsl.dataflow.dflow DEBUG: Task 1982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1982 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1982 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1982 has memoization hash 052e010d428b7165dc48bd89b0945273 parsl.dataflow.memoization INFO: Task 1982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1982 try 0 launched on executor multi with executor id 1983 parsl.dataflow.dflow INFO: Standard output for task 1982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/dc1290d0-39d7-4bf1-97fd-3f5de61930de_isrForCrosstalkSources_1125932_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/dc1290d0-39d7-4bf1-97fd-3f5de61930de_isrForCrosstalkSources_1125932_29.stderr parsl.dataflow.dflow DEBUG: Task 1983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1983 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1983 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1983 has memoization hash acc5b79f70393cef5a0dfdd19899b18d parsl.dataflow.memoization INFO: Task 1983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1983 try 0 launched on executor multi with executor id 1984 parsl.dataflow.dflow INFO: Standard output for task 1983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/0ae3014f-555a-43a5-8b48-b2b48b8b1376_isrForCrosstalkSources_1125919_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/0ae3014f-555a-43a5-8b48-b2b48b8b1376_isrForCrosstalkSources_1125919_4.stderr parsl.dataflow.dflow DEBUG: Task 1984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1984 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1984 has memoization hash 7d7fe03d390c1ae15b0ffbb10975f724 parsl.dataflow.memoization INFO: Task 1984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1984 try 0 launched on executor multi with executor id 1985 parsl.dataflow.dflow INFO: Standard output for task 1984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/31bceb9d-0ed1-476d-8d9f-a2d342f6d533_isrForCrosstalkSources_1125900_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/31bceb9d-0ed1-476d-8d9f-a2d342f6d533_isrForCrosstalkSources_1125900_13.stderr parsl.dataflow.dflow DEBUG: Task 1985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1985 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1985 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1985 has memoization hash d228f51e58610ff718871585ddfe1a7a parsl.dataflow.memoization INFO: Task 1985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1985 try 0 launched on executor multi with executor id 1986 parsl.dataflow.dflow INFO: Standard output for task 1985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/7d1ee071-be62-423d-8260-a7da86badea0_isrForCrosstalkSources_1125891_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/7d1ee071-be62-423d-8260-a7da86badea0_isrForCrosstalkSources_1125891_10.stderr parsl.dataflow.dflow DEBUG: Task 1986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1986 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1986 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1986 has memoization hash f53b9638dd9d42b6af4e265a8fc8432f parsl.dataflow.memoization INFO: Task 1986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1986 try 0 launched on executor multi with executor id 1987 parsl.dataflow.dflow INFO: Standard output for task 1986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/a9317c72-079a-40f9-9bb1-65ef93e448b9_isrForCrosstalkSources_1125897_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/a9317c72-079a-40f9-9bb1-65ef93e448b9_isrForCrosstalkSources_1125897_18.stderr parsl.dataflow.dflow DEBUG: Task 1987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1987 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1987 has memoization hash 675ab75493655d946d917dc076a6d64a parsl.dataflow.memoization INFO: Task 1987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1987 try 0 launched on executor multi with executor id 1988 parsl.dataflow.dflow INFO: Standard output for task 1987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/2e90a15f-56ee-4abe-bb89-f313d11cfab0_isrForCrosstalkSources_1125907_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/2e90a15f-56ee-4abe-bb89-f313d11cfab0_isrForCrosstalkSources_1125907_1.stderr parsl.dataflow.dflow DEBUG: Task 1988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1988 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1988 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1988 has memoization hash 2c90338b43c60e1e0dddf5548b58afe5 parsl.dataflow.memoization INFO: Task 1988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1988 try 0 launched on executor multi with executor id 1989 parsl.dataflow.dflow INFO: Standard output for task 1988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/6790843a-8a5d-4447-9735-38c7c36e2e5c_isrForCrosstalkSources_1125941_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/6790843a-8a5d-4447-9735-38c7c36e2e5c_isrForCrosstalkSources_1125941_43.stderr parsl.dataflow.dflow DEBUG: Task 1989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1989 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1989 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1989 has memoization hash 905b5a5c5285e71bc62eee3d616cbab1 parsl.dataflow.memoization INFO: Task 1989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1989 try 0 launched on executor multi with executor id 1990 parsl.dataflow.dflow INFO: Standard output for task 1989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/00b2e2ff-785b-4026-a8c5-7937d33d6f7d_isrForCrosstalkSources_1125927_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/00b2e2ff-785b-4026-a8c5-7937d33d6f7d_isrForCrosstalkSources_1125927_56.stderr parsl.dataflow.dflow DEBUG: Task 1990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1990 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1990 has memoization hash fe8effbdbb09dff13df9f78a673d5a2f parsl.dataflow.memoization INFO: Task 1990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1990 try 0 launched on executor multi with executor id 1991 parsl.dataflow.dflow INFO: Standard output for task 1990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/9c4f426d-2e66-49d3-8475-df1ccb25e1f1_isrForCrosstalkSources_1125942_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/9c4f426d-2e66-49d3-8475-df1ccb25e1f1_isrForCrosstalkSources_1125942_44.stderr parsl.dataflow.dflow DEBUG: Task 1991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1991 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1991 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1991 has memoization hash 1af090844672e3d9a8e8031213ce5ea5 parsl.dataflow.memoization INFO: Task 1991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1991 try 0 launched on executor multi with executor id 1992 parsl.dataflow.dflow INFO: Standard output for task 1991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/e7ec964c-a391-4137-b1a9-e6f20b1c86be_isrForCrosstalkSources_1125905_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/e7ec964c-a391-4137-b1a9-e6f20b1c86be_isrForCrosstalkSources_1125905_24.stderr parsl.dataflow.dflow DEBUG: Task 1992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1992 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1992 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1992 has memoization hash 16704eba95355f46547f923edb3be3dd parsl.dataflow.memoization INFO: Task 1992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1992 try 0 launched on executor multi with executor id 1993 parsl.dataflow.dflow INFO: Standard output for task 1992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/4c668d50-56f8-4f77-af65-51531032d4e6_isrForCrosstalkSources_1125890_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/4c668d50-56f8-4f77-af65-51531032d4e6_isrForCrosstalkSources_1125890_30.stderr parsl.dataflow.dflow DEBUG: Task 1993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1993 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1993 has memoization hash 68bc0210759e0aefcb9e4a5ad1eee94f parsl.dataflow.memoization INFO: Task 1993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1993 try 0 launched on executor multi with executor id 1994 parsl.dataflow.dflow INFO: Standard output for task 1993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/4d69a4ed-a53e-4668-b311-2c26671e5d5f_isrForCrosstalkSources_1125921_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/4d69a4ed-a53e-4668-b311-2c26671e5d5f_isrForCrosstalkSources_1125921_13.stderr parsl.dataflow.dflow DEBUG: Task 1994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1994 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1994 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1994 has memoization hash 459d9b2cd0911d9f28a2ee309a7274fb parsl.dataflow.memoization INFO: Task 1994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aa020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1994 try 0 launched on executor multi with executor id 1995 parsl.dataflow.dflow INFO: Standard output for task 1994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/27979002-f4a1-4985-947e-b59ac3268482_isrForCrosstalkSources_1125907_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/27979002-f4a1-4985-947e-b59ac3268482_isrForCrosstalkSources_1125907_36.stderr parsl.dataflow.dflow DEBUG: Task 1995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1995 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1995 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1995 has memoization hash 8c393279aed869c4ca42a20bd3e8642f parsl.dataflow.memoization INFO: Task 1995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aa0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1995 try 0 launched on executor multi with executor id 1996 parsl.dataflow.dflow INFO: Standard output for task 1995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/0e1f3798-9291-4504-beec-cafced98e88f_isrForCrosstalkSources_1125935_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/0e1f3798-9291-4504-beec-cafced98e88f_isrForCrosstalkSources_1125935_8.stderr parsl.dataflow.dflow DEBUG: Task 1996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1996 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1996 has memoization hash 61340d28926ac14c6aec068149335515 parsl.dataflow.memoization INFO: Task 1996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aa160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1996 try 0 launched on executor multi with executor id 1997 parsl.dataflow.dflow INFO: Standard output for task 1996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/93fc0c6e-725c-4981-b407-0d5ff6ef4485_isrForCrosstalkSources_1125894_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/93fc0c6e-725c-4981-b407-0d5ff6ef4485_isrForCrosstalkSources_1125894_10.stderr parsl.dataflow.dflow DEBUG: Task 1997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1997 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1997 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1997 has memoization hash 9a2aeaa4af1d30824669c64517a80c33 parsl.dataflow.memoization INFO: Task 1997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aa2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1997 try 0 launched on executor multi with executor id 1998 parsl.dataflow.dflow INFO: Standard output for task 1997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/20b64011-a5a6-46af-8acf-317d17f45b34_isrForCrosstalkSources_1125906_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/20b64011-a5a6-46af-8acf-317d17f45b34_isrForCrosstalkSources_1125906_25.stderr parsl.dataflow.dflow DEBUG: Task 1998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1998 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1998 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1998 has memoization hash 5c98f408210545acaa4cf05e3ab5323a parsl.dataflow.memoization INFO: Task 1998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aa200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1998 try 0 launched on executor multi with executor id 1999 parsl.dataflow.dflow INFO: Standard output for task 1998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/bc4c2c09-bfac-4aff-b11e-5450f2183889_isrForCrosstalkSources_1125898_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/bc4c2c09-bfac-4aff-b11e-5450f2183889_isrForCrosstalkSources_1125898_20.stderr parsl.dataflow.dflow DEBUG: Task 1999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1999 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1999 has memoization hash bae8b009984e3e22e9851db2cf21675a parsl.dataflow.memoization INFO: Task 1999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aa340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1999 try 0 launched on executor multi with executor id 2000 parsl.dataflow.dflow INFO: Standard output for task 1999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/f566a136-7762-459d-825b-c72cbf788394_isrForCrosstalkSources_1125902_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/f566a136-7762-459d-825b-c72cbf788394_isrForCrosstalkSources_1125902_57.stderr parsl.dataflow.dflow DEBUG: Task 2000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2000 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2000 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2000 has memoization hash e1c546edbd485d803bede964a000a5ec parsl.dataflow.memoization INFO: Task 2000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aa3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2000 try 0 launched on executor multi with executor id 2001 parsl.dataflow.dflow INFO: Standard output for task 2000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/2429dfc1-5725-4df9-9059-97bf883f20e9_isrForCrosstalkSources_1125899_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/2429dfc1-5725-4df9-9059-97bf883f20e9_isrForCrosstalkSources_1125899_37.stderr parsl.dataflow.dflow DEBUG: Task 2001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2001 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2001 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2001 has memoization hash a706f4a51df9275e949865641206cf2d parsl.dataflow.memoization INFO: Task 2001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aa480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2001 try 0 launched on executor multi with executor id 2002 parsl.dataflow.dflow INFO: Standard output for task 2001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/89b7b11a-d135-486e-9758-182e6d77ad1f_isrForCrosstalkSources_1125914_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/89b7b11a-d135-486e-9758-182e6d77ad1f_isrForCrosstalkSources_1125914_7.stderr parsl.dataflow.dflow DEBUG: Task 2002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2002 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2002 has memoization hash 680a939e3d1c424b63541094ba5339b7 parsl.dataflow.memoization INFO: Task 2002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aa5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2002 try 0 launched on executor multi with executor id 2003 parsl.dataflow.dflow INFO: Standard output for task 2002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/b933cb4e-a443-406f-8073-a1299fab2d16_isrForCrosstalkSources_1125943_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/b933cb4e-a443-406f-8073-a1299fab2d16_isrForCrosstalkSources_1125943_61.stderr parsl.dataflow.dflow DEBUG: Task 2003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2003 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2003 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2003 has memoization hash 90cb72d9a15c7ab140d758a5fd4d2557 parsl.dataflow.memoization INFO: Task 2003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aa520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2003 try 0 launched on executor multi with executor id 2004 parsl.dataflow.dflow INFO: Standard output for task 2003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/ca9dc70b-63e1-4466-8327-de3fae2ecd5b_isrForCrosstalkSources_1125907_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/ca9dc70b-63e1-4466-8327-de3fae2ecd5b_isrForCrosstalkSources_1125907_41.stderr parsl.dataflow.dflow DEBUG: Task 2004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2004 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2004 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2004 has memoization hash bf8f10c44450c983441ed450d771be38 parsl.dataflow.memoization INFO: Task 2004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aa660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2004 try 0 launched on executor multi with executor id 2005 parsl.dataflow.dflow INFO: Standard output for task 2004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/4864cb05-ebee-48da-9799-e1cbf939c9df_isrForCrosstalkSources_1125936_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/4864cb05-ebee-48da-9799-e1cbf939c9df_isrForCrosstalkSources_1125936_53.stderr parsl.dataflow.dflow DEBUG: Task 2005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2005 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2005 has memoization hash ab07a2efdfea8af469f32636bf0d72de parsl.dataflow.memoization INFO: Task 2005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aa700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2005 try 0 launched on executor multi with executor id 2006 parsl.dataflow.dflow INFO: Standard output for task 2005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d33894fc-dca6-4d19-8307-afc8c1e287c5_isrForCrosstalkSources_1125905_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d33894fc-dca6-4d19-8307-afc8c1e287c5_isrForCrosstalkSources_1125905_22.stderr parsl.dataflow.dflow DEBUG: Task 2006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2006 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2006 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2006 has memoization hash e8d0a168df31a637cbfc0072e6e55b41 parsl.dataflow.memoization INFO: Task 2006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aa7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2006 try 0 launched on executor multi with executor id 2007 parsl.dataflow.dflow INFO: Standard output for task 2006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/e53405a9-54f7-4a03-81fa-0dc677356073_isrForCrosstalkSources_1125936_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/e53405a9-54f7-4a03-81fa-0dc677356073_isrForCrosstalkSources_1125936_36.stderr parsl.dataflow.dflow DEBUG: Task 2007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2007 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2007 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2007 has memoization hash b72abacdbd5e86e9e2c4e5b83d928cf7 parsl.dataflow.memoization INFO: Task 2007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aa8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2007 try 0 launched on executor multi with executor id 2008 parsl.dataflow.dflow INFO: Standard output for task 2007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/d3a9b58e-562d-4dcd-a531-b4b3fb370227_isrForCrosstalkSources_1125947_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/d3a9b58e-562d-4dcd-a531-b4b3fb370227_isrForCrosstalkSources_1125947_8.stderr parsl.dataflow.dflow DEBUG: Task 2008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2008 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2008 has memoization hash a0e8755b5a2a71cbf5f1bcda0414934c parsl.dataflow.memoization INFO: Task 2008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aa840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2008 try 0 launched on executor multi with executor id 2009 parsl.dataflow.dflow INFO: Standard output for task 2008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/a0ed7748-3a01-4704-a78f-a39203013884_isrForCrosstalkSources_1125898_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/a0ed7748-3a01-4704-a78f-a39203013884_isrForCrosstalkSources_1125898_53.stderr parsl.dataflow.dflow DEBUG: Task 2009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2009 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2009 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2009 has memoization hash 9d21b055137aa36c670642c763413fc6 parsl.dataflow.memoization INFO: Task 2009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aa980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2009 try 0 launched on executor multi with executor id 2010 parsl.dataflow.dflow INFO: Standard output for task 2009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ef782dee-1132-4bf0-bd82-8fe2b765b111_isrForCrosstalkSources_1125918_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ef782dee-1132-4bf0-bd82-8fe2b765b111_isrForCrosstalkSources_1125918_31.stderr parsl.dataflow.dflow DEBUG: Task 2010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2010 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2010 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2010 has memoization hash f77eb554ac474859533c552b2153d12f parsl.dataflow.memoization INFO: Task 2010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2010 try 0 launched on executor multi with executor id 2011 parsl.dataflow.dflow INFO: Standard output for task 2010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/8fc290c4-dcb3-47b1-b031-d9d04f9c9320_isrForCrosstalkSources_1125889_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/8fc290c4-dcb3-47b1-b031-d9d04f9c9320_isrForCrosstalkSources_1125889_59.stderr parsl.dataflow.dflow DEBUG: Task 2011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2011 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2011 has memoization hash 114766df6370a24caf2a6a4b6287580d parsl.dataflow.memoization INFO: Task 2011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2011 try 0 launched on executor multi with executor id 2012 parsl.dataflow.dflow INFO: Standard output for task 2011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/71a9d350-1fbc-4cb2-a4fe-55c287ea4b24_isrForCrosstalkSources_1125893_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/71a9d350-1fbc-4cb2-a4fe-55c287ea4b24_isrForCrosstalkSources_1125893_32.stderr parsl.dataflow.dflow DEBUG: Task 2012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2012 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2012 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2012 has memoization hash 1830479461c485c0ed7aeb98ed0c269c parsl.dataflow.memoization INFO: Task 2012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2012 try 0 launched on executor multi with executor id 2013 parsl.dataflow.dflow INFO: Standard output for task 2012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/f56840f4-feb0-44c9-9fcc-6c21cca82a4b_isrForCrosstalkSources_1125892_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/f56840f4-feb0-44c9-9fcc-6c21cca82a4b_isrForCrosstalkSources_1125892_32.stderr parsl.dataflow.dflow DEBUG: Task 2013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2013 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2013 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2013 has memoization hash ef9724f340d635420cf6d48b8e3a1916 parsl.dataflow.memoization INFO: Task 2013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2013 try 0 launched on executor multi with executor id 2014 parsl.dataflow.dflow INFO: Standard output for task 2013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/e7fd03bf-72d4-4238-8c43-f68799863ed9_isrForCrosstalkSources_1125896_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/e7fd03bf-72d4-4238-8c43-f68799863ed9_isrForCrosstalkSources_1125896_6.stderr parsl.dataflow.dflow DEBUG: Task 2014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2014 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2014 has memoization hash fb83da373b5e01f4797332ac0ddc379a parsl.dataflow.memoization INFO: Task 2014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2014 try 0 launched on executor multi with executor id 2015 parsl.dataflow.dflow INFO: Standard output for task 2014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/0625331c-6e19-4ecd-964e-9348bdb73a6a_isrForCrosstalkSources_1125932_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/0625331c-6e19-4ecd-964e-9348bdb73a6a_isrForCrosstalkSources_1125932_59.stderr parsl.dataflow.dflow DEBUG: Task 2015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2015 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2015 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2015 has memoization hash 311dc7c8e406865eb910a7cbd8df8c03 parsl.dataflow.memoization INFO: Task 2015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2015 try 0 launched on executor multi with executor id 2016 parsl.dataflow.dflow INFO: Standard output for task 2015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/72402885-ef99-45d5-b313-741c0d85a210_isrForCrosstalkSources_1125940_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/72402885-ef99-45d5-b313-741c0d85a210_isrForCrosstalkSources_1125940_30.stderr parsl.dataflow.dflow DEBUG: Task 2016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2016 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2016 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2016 has memoization hash 2044fbfbe3848a1a6c52ce695ef60ad7 parsl.dataflow.memoization INFO: Task 2016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2016 try 0 launched on executor multi with executor id 2017 parsl.dataflow.dflow INFO: Standard output for task 2016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/7b0c44ae-9635-4d20-ac02-e4d7c824e7b7_isrForCrosstalkSources_1125896_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/7b0c44ae-9635-4d20-ac02-e4d7c824e7b7_isrForCrosstalkSources_1125896_59.stderr parsl.dataflow.dflow DEBUG: Task 2017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2017 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2017 has memoization hash 26b083238b356db69ab36646479843b3 parsl.dataflow.memoization INFO: Task 2017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aaf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2017 try 0 launched on executor multi with executor id 2018 parsl.dataflow.dflow INFO: Standard output for task 2017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/3f36ffb5-14e0-42bd-af18-f5c73c1b1ade_isrForCrosstalkSources_1125921_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/3f36ffb5-14e0-42bd-af18-f5c73c1b1ade_isrForCrosstalkSources_1125921_35.stderr parsl.dataflow.dflow DEBUG: Task 2018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2018 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2018 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2018 has memoization hash e769380b4d188d255652a821a3a650d1 parsl.dataflow.memoization INFO: Task 2018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2018 try 0 launched on executor multi with executor id 2019 parsl.dataflow.dflow INFO: Standard output for task 2018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/db88680f-5e1c-403a-9ae6-58af76ee0ed8_isrForCrosstalkSources_1125899_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/db88680f-5e1c-403a-9ae6-58af76ee0ed8_isrForCrosstalkSources_1125899_21.stderr parsl.dataflow.dflow DEBUG: Task 2019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2019 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2019 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2019 has memoization hash c751a58ba621ab794b28660b681e961b parsl.dataflow.memoization INFO: Task 2019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2019 try 0 launched on executor multi with executor id 2020 parsl.dataflow.dflow INFO: Standard output for task 2019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/b8f84b9b-6db4-428b-b260-1173b7a41c5b_isrForCrosstalkSources_1125939_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/b8f84b9b-6db4-428b-b260-1173b7a41c5b_isrForCrosstalkSources_1125939_60.stderr parsl.dataflow.dflow DEBUG: Task 2020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2020 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2020 has memoization hash e23e276fc7fcf81fbe430f083dc18819 parsl.dataflow.memoization INFO: Task 2020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399ab060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2020 try 0 launched on executor multi with executor id 2021 parsl.dataflow.dflow INFO: Standard output for task 2020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/f437ff3f-3e7b-4e5c-8fa3-76c87fe3fdc6_isrForCrosstalkSources_1125919_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/f437ff3f-3e7b-4e5c-8fa3-76c87fe3fdc6_isrForCrosstalkSources_1125919_38.stderr parsl.dataflow.dflow DEBUG: Task 2021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2021 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2021 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2021 has memoization hash 46c396dc5665d82321c780103d8a9b50 parsl.dataflow.memoization INFO: Task 2021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399ab100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2021 try 0 launched on executor multi with executor id 2022 parsl.dataflow.dflow INFO: Standard output for task 2021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8b742e02-811f-4d7a-aa9d-83a6fb37a4f1_isrForCrosstalkSources_1125891_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8b742e02-811f-4d7a-aa9d-83a6fb37a4f1_isrForCrosstalkSources_1125891_28.stderr parsl.dataflow.dflow DEBUG: Task 2022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2022 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2022 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2022 has memoization hash 651d79f0fa2f334b13b73fc848716c8b parsl.dataflow.memoization INFO: Task 2022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399ab240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2022 try 0 launched on executor multi with executor id 2023 parsl.dataflow.dflow INFO: Standard output for task 2022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/973c9c40-2483-4386-bf1c-bbd8c7133490_isrForCrosstalkSources_1125901_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/973c9c40-2483-4386-bf1c-bbd8c7133490_isrForCrosstalkSources_1125901_30.stderr parsl.dataflow.dflow DEBUG: Task 2023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2023 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2023 has memoization hash e228ef62583b69ff1871bb78307e9541 parsl.dataflow.memoization INFO: Task 2023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399ab1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2023 try 0 launched on executor multi with executor id 2024 parsl.dataflow.dflow INFO: Standard output for task 2023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/9579481b-836b-4233-b672-f489f8f53f2b_isrForCrosstalkSources_1125904_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/9579481b-836b-4233-b672-f489f8f53f2b_isrForCrosstalkSources_1125904_34.stderr parsl.dataflow.dflow DEBUG: Task 2024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2024 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2024 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2024 has memoization hash 1d0a4eadadaca20b542348abbd279b08 parsl.dataflow.memoization INFO: Task 2024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399ab2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2024 try 0 launched on executor multi with executor id 2025 parsl.dataflow.dflow INFO: Standard output for task 2024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/52e091b1-2c74-4877-9910-d3c07c846d1d_isrForCrosstalkSources_1125894_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/52e091b1-2c74-4877-9910-d3c07c846d1d_isrForCrosstalkSources_1125894_31.stderr parsl.dataflow.dflow DEBUG: Task 2025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2025 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2025 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2025 has memoization hash 9bed466d8a2ad60e86775591f5edd9f8 parsl.dataflow.memoization INFO: Task 2025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399ab380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2025 try 0 launched on executor multi with executor id 2026 parsl.dataflow.dflow INFO: Standard output for task 2025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/c37e41ab-6872-4ad1-975a-8db50e2dfa32_isrForCrosstalkSources_1125923_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/c37e41ab-6872-4ad1-975a-8db50e2dfa32_isrForCrosstalkSources_1125923_6.stderr parsl.dataflow.dflow DEBUG: Task 2026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2026 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2026 has memoization hash 25ed1468cabbba48ad00a8d88acba50e parsl.dataflow.memoization INFO: Task 2026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399ab420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2026 try 0 launched on executor multi with executor id 2027 parsl.dataflow.dflow INFO: Standard output for task 2026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/0c9d69ba-de35-4765-b9a9-50263915b5c1_isrForCrosstalkSources_1125903_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/0c9d69ba-de35-4765-b9a9-50263915b5c1_isrForCrosstalkSources_1125903_46.stderr parsl.dataflow.dflow DEBUG: Task 2027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2027 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2027 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2027 has memoization hash 064a2a21c3b042bbbee2593d67b3f27f parsl.dataflow.memoization INFO: Task 2027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399ab560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2027 try 0 launched on executor multi with executor id 2028 parsl.dataflow.dflow INFO: Standard output for task 2027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0d109018-546a-4bca-9f3f-ac5f3f6c929b_isrForCrosstalkSources_1125904_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0d109018-546a-4bca-9f3f-ac5f3f6c929b_isrForCrosstalkSources_1125904_52.stderr parsl.dataflow.dflow DEBUG: Task 2028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2028 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2028 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2028 has memoization hash 105e4b1b72a81b7d8ab411a84ac65331 parsl.dataflow.memoization INFO: Task 2028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399ab4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2028 try 0 launched on executor multi with executor id 2029 parsl.dataflow.dflow INFO: Standard output for task 2028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/88b435c7-04a3-4508-9859-cfaa07b0688e_isrForCrosstalkSources_1125892_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/88b435c7-04a3-4508-9859-cfaa07b0688e_isrForCrosstalkSources_1125892_54.stderr parsl.dataflow.dflow DEBUG: Task 2029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2029 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2029 has memoization hash b38b36d150e386c70f4659818aab8449 parsl.dataflow.memoization INFO: Task 2029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399ab600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2029 try 0 launched on executor multi with executor id 2030 parsl.dataflow.dflow INFO: Standard output for task 2029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/d5ad1027-3646-4e33-be2a-ddb3960c3f22_isrForCrosstalkSources_1125910_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/d5ad1027-3646-4e33-be2a-ddb3960c3f22_isrForCrosstalkSources_1125910_33.stderr parsl.dataflow.dflow DEBUG: Task 2030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2030 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2030 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2030 has memoization hash 9aac62686e9e3acbb6ed4d3ba7063399 parsl.dataflow.memoization INFO: Task 2030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399ab6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2030 try 0 launched on executor multi with executor id 2031 parsl.dataflow.dflow INFO: Standard output for task 2030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/b0238dea-5353-42bb-b798-64e98757daec_isrForCrosstalkSources_1125941_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/b0238dea-5353-42bb-b798-64e98757daec_isrForCrosstalkSources_1125941_20.stderr parsl.dataflow.dflow DEBUG: Task 2031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2031 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2031 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2031 has memoization hash 8be64bbd08d3a48b55ea4385b54f5b74 parsl.dataflow.memoization INFO: Task 2031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399ab740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2031 try 0 launched on executor multi with executor id 2032 parsl.dataflow.dflow INFO: Standard output for task 2031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/3d101bb2-d9a1-4bab-a811-41ead43fd38c_isrForCrosstalkSources_1125911_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/3d101bb2-d9a1-4bab-a811-41ead43fd38c_isrForCrosstalkSources_1125911_44.stderr parsl.dataflow.dflow DEBUG: Task 2032 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2032 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2032 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2032 has memoization hash 66da70d2829ed3cb5a2c5d7b62b37daa parsl.dataflow.memoization INFO: Task 2032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399ab880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2032 try 0 launched on executor multi with executor id 2033 parsl.dataflow.dflow INFO: Standard output for task 2032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/e5fd10f4-aeb2-44b3-ad8d-4d0171346b25_isrForCrosstalkSources_1125924_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/e5fd10f4-aeb2-44b3-ad8d-4d0171346b25_isrForCrosstalkSources_1125924_46.stderr parsl.dataflow.dflow DEBUG: Task 2033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2033 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2033 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2033 has memoization hash 47f8399af60cb54e74daa789f4bb3aed parsl.dataflow.memoization INFO: Task 2033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399ab7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2033 try 0 launched on executor multi with executor id 2034 parsl.dataflow.dflow INFO: Standard output for task 2033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/75cede3c-12fe-4acb-9843-7178bfad1a82_isrForCrosstalkSources_1125910_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/75cede3c-12fe-4acb-9843-7178bfad1a82_isrForCrosstalkSources_1125910_48.stderr parsl.dataflow.dflow DEBUG: Task 2034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2034 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2034 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2034 has memoization hash 7b7a20f6da8b283d2e0de3b312b76708 parsl.dataflow.memoization INFO: Task 2034 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399ab920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2034 try 0 launched on executor multi with executor id 2035 parsl.dataflow.dflow INFO: Standard output for task 2034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/fe1ca8a7-cbfa-4197-b277-99c7e1d74400_isrForCrosstalkSources_1125931_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/fe1ca8a7-cbfa-4197-b277-99c7e1d74400_isrForCrosstalkSources_1125931_41.stderr parsl.dataflow.dflow DEBUG: Task 2035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2035 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2035 has memoization hash fe842eb08f0bcc74c055b0fea8011868 parsl.dataflow.memoization INFO: Task 2035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399ab9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2035 try 0 launched on executor multi with executor id 2036 parsl.dataflow.dflow INFO: Standard output for task 2035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/17fe8818-a132-453d-a8f6-d84b33a3270a_isrForCrosstalkSources_1125893_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/17fe8818-a132-453d-a8f6-d84b33a3270a_isrForCrosstalkSources_1125893_4.stderr parsl.dataflow.dflow DEBUG: Task 2036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2036 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2036 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2036 has memoization hash bb37333f676141ea7f9743d1ae8c93c2 parsl.dataflow.memoization INFO: Task 2036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399aba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2036 try 0 launched on executor multi with executor id 2037 parsl.dataflow.dflow INFO: Standard output for task 2036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/b7d34be0-9951-4195-bc47-83fd660d6422_isrForCrosstalkSources_1125915_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/b7d34be0-9951-4195-bc47-83fd660d6422_isrForCrosstalkSources_1125915_8.stderr parsl.dataflow.dflow DEBUG: Task 2037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2037 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2037 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2037 has memoization hash 8047fc17626f86918aa7c06561dd8e1d parsl.dataflow.memoization INFO: Task 2037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399abba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2037 try 0 launched on executor multi with executor id 2038 parsl.dataflow.dflow INFO: Standard output for task 2037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/89774efb-5968-4f7c-930a-fb80d4d114fa_isrForCrosstalkSources_1125904_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/89774efb-5968-4f7c-930a-fb80d4d114fa_isrForCrosstalkSources_1125904_35.stderr parsl.dataflow.dflow DEBUG: Task 2038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2038 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2038 has memoization hash b531f310518489fc35ad9e4063d8bf20 parsl.dataflow.memoization INFO: Task 2038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399abb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2038 try 0 launched on executor multi with executor id 2039 parsl.dataflow.dflow INFO: Standard output for task 2038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/ef4a13e2-b0a9-47e1-9092-7411d62671c1_isrForCrosstalkSources_1125920_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/ef4a13e2-b0a9-47e1-9092-7411d62671c1_isrForCrosstalkSources_1125920_4.stderr parsl.dataflow.dflow DEBUG: Task 2039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2039 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2039 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2039 has memoization hash eb32c8aea45760488e9457799c5b275c parsl.dataflow.memoization INFO: Task 2039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399abc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2039 try 0 launched on executor multi with executor id 2040 parsl.dataflow.dflow INFO: Standard output for task 2039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/437db6b0-a254-4410-8694-3742b9fcb078_isrForCrosstalkSources_1125935_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/437db6b0-a254-4410-8694-3742b9fcb078_isrForCrosstalkSources_1125935_41.stderr parsl.dataflow.dflow DEBUG: Task 2040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2040 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2040 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2040 has memoization hash 98217719b312c4c399839b146f6a49f8 parsl.dataflow.memoization INFO: Task 2040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399abce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2040 try 0 launched on executor multi with executor id 2041 parsl.dataflow.dflow INFO: Standard output for task 2040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/bbbab053-3f7e-4e04-8b62-67c863a180bc_isrForCrosstalkSources_1125894_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/bbbab053-3f7e-4e04-8b62-67c863a180bc_isrForCrosstalkSources_1125894_52.stderr parsl.dataflow.dflow DEBUG: Task 2041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2041 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2041 has memoization hash 4224cbe41bdf9b63fee654baff4fa3c9 parsl.dataflow.memoization INFO: Task 2041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399abd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2041 try 0 launched on executor multi with executor id 2042 parsl.dataflow.dflow INFO: Standard output for task 2041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e484790a-edf0-4bcc-a500-87cadbca1277_isrForCrosstalkSources_1125929_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e484790a-edf0-4bcc-a500-87cadbca1277_isrForCrosstalkSources_1125929_6.stderr parsl.dataflow.dflow DEBUG: Task 2042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2042 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2042 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2042 has memoization hash 1fdb3d01643d2d1515dd6c390301cb35 parsl.dataflow.memoization INFO: Task 2042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399abec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2042 try 0 launched on executor multi with executor id 2043 parsl.dataflow.dflow INFO: Standard output for task 2042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/16e7aebc-d410-4076-b5e6-999499382d5a_isrForCrosstalkSources_1125933_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/16e7aebc-d410-4076-b5e6-999499382d5a_isrForCrosstalkSources_1125933_21.stderr parsl.dataflow.dflow DEBUG: Task 2043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2043 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2043 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2043 has memoization hash 0e9f90d5b68feb5861c0293f210266d5 parsl.dataflow.memoization INFO: Task 2043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399abe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2043 try 0 launched on executor multi with executor id 2044 parsl.dataflow.dflow INFO: Standard output for task 2043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/f655f3c2-9b26-4fee-b7a7-e40f245fb163_isrForCrosstalkSources_1125901_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/f655f3c2-9b26-4fee-b7a7-e40f245fb163_isrForCrosstalkSources_1125901_57.stderr parsl.dataflow.dflow DEBUG: Task 2044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2044 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2044 has memoization hash 35e4fe4961dfa9edfffdf6fdd13c2a97 parsl.dataflow.memoization INFO: Task 2044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399abf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2044 try 0 launched on executor multi with executor id 2045 parsl.dataflow.dflow INFO: Standard output for task 2044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/82fb2b76-f0d5-409f-80eb-2cebbd77cbfd_isrForCrosstalkSources_1125914_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/82fb2b76-f0d5-409f-80eb-2cebbd77cbfd_isrForCrosstalkSources_1125914_20.stderr parsl.dataflow.dflow DEBUG: Task 2045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2045 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2045 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2045 has memoization hash a8df0a23890320de2da1b86240262975 parsl.dataflow.memoization INFO: Task 2045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a399a8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2045 try 0 launched on executor multi with executor id 2046 parsl.dataflow.dflow INFO: Standard output for task 2045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/f6caac42-fcef-4f20-b166-169aa52a77d7_isrForCrosstalkSources_1125915_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/f6caac42-fcef-4f20-b166-169aa52a77d7_isrForCrosstalkSources_1125915_23.stderr parsl.dataflow.dflow DEBUG: Task 2046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2046 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2046 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2046 has memoization hash c2567de99283ecf1b05880353debdf08 parsl.dataflow.memoization INFO: Task 2046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398180e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2046 try 0 launched on executor multi with executor id 2047 parsl.dataflow.dflow INFO: Standard output for task 2046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/b3f5d013-79ca-4d24-9549-92f51e5c47c9_isrForCrosstalkSources_1125909_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/b3f5d013-79ca-4d24-9549-92f51e5c47c9_isrForCrosstalkSources_1125909_11.stderr parsl.dataflow.dflow DEBUG: Task 2047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2047 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2047 has memoization hash 0c36fd63f2073a3b7bd45a41a9193709 parsl.dataflow.memoization INFO: Task 2047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2047 try 0 launched on executor multi with executor id 2048 parsl.dataflow.dflow INFO: Standard output for task 2047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/6664fc09-85d2-4e21-9ca0-549767597ff5_isrForCrosstalkSources_1125934_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/6664fc09-85d2-4e21-9ca0-549767597ff5_isrForCrosstalkSources_1125934_53.stderr parsl.dataflow.dflow DEBUG: Task 2048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2048 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2048 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2048 has memoization hash 4ef560034ded3bc53cdc5fe696b37679 parsl.dataflow.memoization INFO: Task 2048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2048 try 0 launched on executor multi with executor id 2049 parsl.dataflow.dflow INFO: Standard output for task 2048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/7f871063-4d26-4359-bfeb-3b609ce53633_isrForCrosstalkSources_1125931_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/7f871063-4d26-4359-bfeb-3b609ce53633_isrForCrosstalkSources_1125931_17.stderr parsl.dataflow.dflow DEBUG: Task 2049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2049 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2049 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2049 has memoization hash 4e145cbc0ee8d164e625def4bbb3ae26 parsl.dataflow.memoization INFO: Task 2049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398182c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2049 try 0 launched on executor multi with executor id 2050 parsl.dataflow.dflow INFO: Standard output for task 2049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/bf7a9259-4c89-48fc-9e16-bd9f2769890d_isrForCrosstalkSources_1125946_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/bf7a9259-4c89-48fc-9e16-bd9f2769890d_isrForCrosstalkSources_1125946_47.stderr parsl.dataflow.dflow DEBUG: Task 2050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2050 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2050 has memoization hash 21bfb4acb1a523720e4916c7cc95201a parsl.dataflow.memoization INFO: Task 2050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2050 try 0 launched on executor multi with executor id 2051 parsl.dataflow.dflow INFO: Standard output for task 2050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d7a67265-743c-4d62-b9ae-a3d20ecc25da_isrForCrosstalkSources_1125890_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d7a67265-743c-4d62-b9ae-a3d20ecc25da_isrForCrosstalkSources_1125890_53.stderr parsl.dataflow.dflow DEBUG: Task 2051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2051 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2051 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2051 has memoization hash bf569dc2eb6fbbf4d57beeab5355a490 parsl.dataflow.memoization INFO: Task 2051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2051 try 0 launched on executor multi with executor id 2052 parsl.dataflow.dflow INFO: Standard output for task 2051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/c2579187-2861-44e0-b9d2-9b32b5efc9f0_isrForCrosstalkSources_1125903_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/c2579187-2861-44e0-b9d2-9b32b5efc9f0_isrForCrosstalkSources_1125903_40.stderr parsl.dataflow.dflow DEBUG: Task 2052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2052 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2052 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2052 has memoization hash 96bd9573bda854d210c908df25008a2c parsl.dataflow.memoization INFO: Task 2052 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2052 try 0 launched on executor multi with executor id 2053 parsl.dataflow.dflow INFO: Standard output for task 2052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/a728a166-3c23-44a2-9a11-23ae02196be9_isrForCrosstalkSources_1125907_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/a728a166-3c23-44a2-9a11-23ae02196be9_isrForCrosstalkSources_1125907_60.stderr parsl.dataflow.dflow DEBUG: Task 2053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2053 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2053 has memoization hash c9f2e26106dc5b2cdb0895693e16dff9 parsl.dataflow.memoization INFO: Task 2053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398184a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2053 try 0 launched on executor multi with executor id 2054 parsl.dataflow.dflow INFO: Standard output for task 2053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/e8cc4d04-7358-4568-aa15-460cb97ba5aa_isrForCrosstalkSources_1125947_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/e8cc4d04-7358-4568-aa15-460cb97ba5aa_isrForCrosstalkSources_1125947_46.stderr parsl.dataflow.dflow DEBUG: Task 2054 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2054 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2054 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2054 has memoization hash 0bb4fedb3e11f703096e994240a41502 parsl.dataflow.memoization INFO: Task 2054 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398185e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2054 try 0 launched on executor multi with executor id 2055 parsl.dataflow.dflow INFO: Standard output for task 2054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/30ff5c45-c0e0-4ae4-990c-9f926c3659f9_isrForCrosstalkSources_1125901_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/30ff5c45-c0e0-4ae4-990c-9f926c3659f9_isrForCrosstalkSources_1125901_32.stderr parsl.dataflow.dflow DEBUG: Task 2055 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2055 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2055 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2055 has memoization hash 3f686dedae8a88b56db265cb82435c5f parsl.dataflow.memoization INFO: Task 2055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2055 try 0 launched on executor multi with executor id 2056 parsl.dataflow.dflow INFO: Standard output for task 2055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/9bf7f4fa-8afb-4b3f-a6f7-c524602a1705_isrForCrosstalkSources_1125913_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/9bf7f4fa-8afb-4b3f-a6f7-c524602a1705_isrForCrosstalkSources_1125913_55.stderr parsl.dataflow.dflow DEBUG: Task 2056 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2056 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2056 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2056 has memoization hash 8a1da0638bc085e9a99fab847b88e030 parsl.dataflow.memoization INFO: Task 2056 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2056 try 0 launched on executor multi with executor id 2057 parsl.dataflow.dflow INFO: Standard output for task 2056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/d7067b4b-d0ba-4c0c-bb7b-a8c49636f094_isrForCrosstalkSources_1125900_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/d7067b4b-d0ba-4c0c-bb7b-a8c49636f094_isrForCrosstalkSources_1125900_52.stderr parsl.dataflow.dflow DEBUG: Task 2057 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2057 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2057 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2057 has memoization hash d128a0b16b82355efac50d2c4218e1c1 parsl.dataflow.memoization INFO: Task 2057 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2057 try 0 launched on executor multi with executor id 2058 parsl.dataflow.dflow INFO: Standard output for task 2057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/4e5dfb2f-2ae1-428c-8971-b2d5bb457cb9_isrForCrosstalkSources_1125894_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/4e5dfb2f-2ae1-428c-8971-b2d5bb457cb9_isrForCrosstalkSources_1125894_37.stderr parsl.dataflow.dflow DEBUG: Task 2058 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2058 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2058 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2058 has memoization hash 1ad8941a16adf5ed3e7e758503b0938b parsl.dataflow.memoization INFO: Task 2058 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398187c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2058 try 0 launched on executor multi with executor id 2059 parsl.dataflow.dflow INFO: Standard output for task 2058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/12c1ce46-b3b0-492a-81f9-3fcb60b187d8_isrForCrosstalkSources_1125899_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/12c1ce46-b3b0-492a-81f9-3fcb60b187d8_isrForCrosstalkSources_1125899_39.stderr parsl.dataflow.dflow DEBUG: Task 2059 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2059 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2059 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2059 has memoization hash 9d4e81604e9ddc5098e7add28f58b7c3 parsl.dataflow.memoization INFO: Task 2059 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2059 try 0 launched on executor multi with executor id 2060 parsl.dataflow.dflow INFO: Standard output for task 2059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/e36fb971-4364-48e4-8494-df6db69469d5_isrForCrosstalkSources_1125891_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/e36fb971-4364-48e4-8494-df6db69469d5_isrForCrosstalkSources_1125891_22.stderr parsl.dataflow.dflow DEBUG: Task 2060 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2060 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2060 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2060 has memoization hash dbe4ee024803f3ef0ce62a8297def6d3 parsl.dataflow.memoization INFO: Task 2060 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398189a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2060 try 0 launched on executor multi with executor id 2061 parsl.dataflow.dflow INFO: Standard output for task 2060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/3dbdcb2a-3e40-4566-b9b7-08775f4385ea_isrForCrosstalkSources_1125903_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/3dbdcb2a-3e40-4566-b9b7-08775f4385ea_isrForCrosstalkSources_1125903_59.stderr parsl.dataflow.dflow DEBUG: Task 2061 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2061 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2061 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2061 has memoization hash 15073426f2c48f392c87da3b450734a7 parsl.dataflow.memoization INFO: Task 2061 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2061 try 0 launched on executor multi with executor id 2062 parsl.dataflow.dflow INFO: Standard output for task 2061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/ac4599a7-61b9-421c-8268-2121bb7a2027_isrForCrosstalkSources_1125920_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/ac4599a7-61b9-421c-8268-2121bb7a2027_isrForCrosstalkSources_1125920_15.stderr parsl.dataflow.dflow DEBUG: Task 2062 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2062 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2062 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2062 has memoization hash c4c844b775f3b5ddd3e421991f738095 parsl.dataflow.memoization INFO: Task 2062 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2062 try 0 launched on executor multi with executor id 2063 parsl.dataflow.dflow INFO: Standard output for task 2062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/12d0ae75-bc60-44af-a081-e5deeda66bb6_isrForCrosstalkSources_1125935_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/12d0ae75-bc60-44af-a081-e5deeda66bb6_isrForCrosstalkSources_1125935_24.stderr parsl.dataflow.dflow DEBUG: Task 2063 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2063 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2063 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2063 has memoization hash fb414f57a75210a96f8fb58a6a0d8a21 parsl.dataflow.memoization INFO: Task 2063 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2063 try 0 launched on executor multi with executor id 2064 parsl.dataflow.dflow INFO: Standard output for task 2063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/cf54d74a-cadc-4043-9ffe-27801c828ee7_isrForCrosstalkSources_1125944_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/cf54d74a-cadc-4043-9ffe-27801c828ee7_isrForCrosstalkSources_1125944_49.stderr parsl.dataflow.dflow DEBUG: Task 2064 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2064 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2064 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2064 has memoization hash 1098c84fd918f4a233f6defb3b5efa5f parsl.dataflow.memoization INFO: Task 2064 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2064 try 0 launched on executor multi with executor id 2065 parsl.dataflow.dflow INFO: Standard output for task 2064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/ffcdeb66-ea74-45a0-b6c6-3317995683f4_isrForCrosstalkSources_1125904_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/ffcdeb66-ea74-45a0-b6c6-3317995683f4_isrForCrosstalkSources_1125904_42.stderr parsl.dataflow.dflow DEBUG: Task 2065 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2065 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2065 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2065 has memoization hash 00b7ea6eeb11cd4bdac903cba8a3fa1a parsl.dataflow.memoization INFO: Task 2065 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2065 try 0 launched on executor multi with executor id 2066 parsl.dataflow.dflow INFO: Standard output for task 2065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/5eac6ba4-1997-4011-90ad-882489271096_isrForCrosstalkSources_1125905_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/5eac6ba4-1997-4011-90ad-882489271096_isrForCrosstalkSources_1125905_61.stderr parsl.dataflow.dflow DEBUG: Task 2066 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2066 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2066 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2066 has memoization hash 2387de0bf78e9e69cea1c09577e2bc60 parsl.dataflow.memoization INFO: Task 2066 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2066 try 0 launched on executor multi with executor id 2067 parsl.dataflow.dflow INFO: Standard output for task 2066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/13acd120-d30b-4708-87e9-87a8627af8f2_isrForCrosstalkSources_1125933_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/13acd120-d30b-4708-87e9-87a8627af8f2_isrForCrosstalkSources_1125933_55.stderr parsl.dataflow.dflow DEBUG: Task 2067 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2067 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2067 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2067 has memoization hash d3c0b5356cc785c7de8299805ec29b75 parsl.dataflow.memoization INFO: Task 2067 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2067 try 0 launched on executor multi with executor id 2068 parsl.dataflow.dflow INFO: Standard output for task 2067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/5d534ab5-620a-4869-8774-485bb4dc6dd9_isrForCrosstalkSources_1125936_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/5d534ab5-620a-4869-8774-485bb4dc6dd9_isrForCrosstalkSources_1125936_30.stderr parsl.dataflow.dflow DEBUG: Task 2068 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2068 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2068 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2068 has memoization hash 0a7e70fa1354f077c2659e0b2b7a7e83 parsl.dataflow.memoization INFO: Task 2068 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2068 try 0 launched on executor multi with executor id 2069 parsl.dataflow.dflow INFO: Standard output for task 2068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/580ce628-f0aa-4520-935a-86a0515de81c_isrForCrosstalkSources_1125923_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/580ce628-f0aa-4520-935a-86a0515de81c_isrForCrosstalkSources_1125923_36.stderr parsl.dataflow.dflow DEBUG: Task 2069 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2069 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2069 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2069 has memoization hash 511358262deb8d5677d96a2c5a8e83ae parsl.dataflow.memoization INFO: Task 2069 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2069 try 0 launched on executor multi with executor id 2070 parsl.dataflow.dflow INFO: Standard output for task 2069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e68d00ab-cd59-42ec-9142-ff5e918cf99c_isrForCrosstalkSources_1125916_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e68d00ab-cd59-42ec-9142-ff5e918cf99c_isrForCrosstalkSources_1125916_27.stderr parsl.dataflow.dflow DEBUG: Task 2070 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2070 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2070 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2070 has memoization hash 316a02ce3bf7c01fae43c858296494aa parsl.dataflow.memoization INFO: Task 2070 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2070 try 0 launched on executor multi with executor id 2071 parsl.dataflow.dflow INFO: Standard output for task 2070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f31bc80f-9f8b-4ce5-8a17-84141dae09d6_isrForCrosstalkSources_1125939_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f31bc80f-9f8b-4ce5-8a17-84141dae09d6_isrForCrosstalkSources_1125939_23.stderr parsl.dataflow.dflow DEBUG: Task 2071 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2071 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2071 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2071 has memoization hash d6c3b4c754968dd3568bc11138547bee parsl.dataflow.memoization INFO: Task 2071 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2071 try 0 launched on executor multi with executor id 2072 parsl.dataflow.dflow INFO: Standard output for task 2071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/b50afede-3498-4fe6-9f78-215d95b971e3_isrForCrosstalkSources_1125900_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/b50afede-3498-4fe6-9f78-215d95b971e3_isrForCrosstalkSources_1125900_46.stderr parsl.dataflow.dflow DEBUG: Task 2072 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2072 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2072 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2072 has memoization hash e0f025dfe76430a419812b907d6950c4 parsl.dataflow.memoization INFO: Task 2072 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398191c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2072 try 0 launched on executor multi with executor id 2073 parsl.dataflow.dflow INFO: Standard output for task 2072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/87e31c24-96b7-4819-826b-9f105fed9059_isrForCrosstalkSources_1125947_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/87e31c24-96b7-4819-826b-9f105fed9059_isrForCrosstalkSources_1125947_6.stderr parsl.dataflow.dflow DEBUG: Task 2073 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2073 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2073 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2073 has memoization hash 328c4ab219b2c53676971f41f77720bc parsl.dataflow.memoization INFO: Task 2073 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2073 try 0 launched on executor multi with executor id 2074 parsl.dataflow.dflow INFO: Standard output for task 2073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/bb335180-7aa1-439e-915e-e43e1327b425_isrForCrosstalkSources_1125889_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/bb335180-7aa1-439e-915e-e43e1327b425_isrForCrosstalkSources_1125889_22.stderr parsl.dataflow.dflow DEBUG: Task 2074 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2074 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2074 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2074 has memoization hash dd393fe4b986a0220f505170040df8ec parsl.dataflow.memoization INFO: Task 2074 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2074 try 0 launched on executor multi with executor id 2075 parsl.dataflow.dflow INFO: Standard output for task 2074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/223f18db-0f63-4e9d-97b9-503994fc3c55_isrForCrosstalkSources_1125918_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/223f18db-0f63-4e9d-97b9-503994fc3c55_isrForCrosstalkSources_1125918_27.stderr parsl.dataflow.dflow DEBUG: Task 2075 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2075 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2075 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2075 has memoization hash 5fc3da458866898211ed913b136e3a9e parsl.dataflow.memoization INFO: Task 2075 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2075 try 0 launched on executor multi with executor id 2076 parsl.dataflow.dflow INFO: Standard output for task 2075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/0e100123-3764-409e-945d-b4c987fd4c53_isrForCrosstalkSources_1125936_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/0e100123-3764-409e-945d-b4c987fd4c53_isrForCrosstalkSources_1125936_35.stderr parsl.dataflow.dflow DEBUG: Task 2076 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2076 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2076 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2076 has memoization hash 992fd9197becb5621467d104fd9e0185 parsl.dataflow.memoization INFO: Task 2076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398193a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2076 try 0 launched on executor multi with executor id 2077 parsl.dataflow.dflow INFO: Standard output for task 2076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/950851fc-5dcd-4c79-8953-99ea2be12d5c_isrForCrosstalkSources_1125941_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/950851fc-5dcd-4c79-8953-99ea2be12d5c_isrForCrosstalkSources_1125941_25.stderr parsl.dataflow.dflow DEBUG: Task 2077 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2077 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2077 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2077 has memoization hash dcbea14622f7b5d5a1b94eaadbb345db parsl.dataflow.memoization INFO: Task 2077 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398194e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2077 try 0 launched on executor multi with executor id 2078 parsl.dataflow.dflow INFO: Standard output for task 2077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/86c06a0c-d8ca-4baa-bbc7-45f1c1d033d6_isrForCrosstalkSources_1125904_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/86c06a0c-d8ca-4baa-bbc7-45f1c1d033d6_isrForCrosstalkSources_1125904_44.stderr parsl.dataflow.dflow DEBUG: Task 2078 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2078 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2078 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2078 has memoization hash e741851885d2c986d3d033df3f6887d3 parsl.dataflow.memoization INFO: Task 2078 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2078 try 0 launched on executor multi with executor id 2079 parsl.dataflow.dflow INFO: Standard output for task 2078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/27144643-844f-4240-b059-b7e6fa459f41_isrForCrosstalkSources_1125902_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/27144643-844f-4240-b059-b7e6fa459f41_isrForCrosstalkSources_1125902_13.stderr parsl.dataflow.dflow DEBUG: Task 2079 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2079 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2079 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2079 has memoization hash 15016aa83b95684b6f2d5fe73d258c56 parsl.dataflow.memoization INFO: Task 2079 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2079 try 0 launched on executor multi with executor id 2080 parsl.dataflow.dflow INFO: Standard output for task 2079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/a38893df-bb6d-4866-a971-0cfb6a964df4_isrForCrosstalkSources_1125920_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/a38893df-bb6d-4866-a971-0cfb6a964df4_isrForCrosstalkSources_1125920_20.stderr parsl.dataflow.dflow DEBUG: Task 2080 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2080 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2080 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2080 has memoization hash 87b3bcc34f8e2e5b1571be07cfd8d2e8 parsl.dataflow.memoization INFO: Task 2080 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2080 try 0 launched on executor multi with executor id 2081 parsl.dataflow.dflow INFO: Standard output for task 2080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/19997bff-55e7-42bd-bf6b-9372c89f7b5f_isrForCrosstalkSources_1125913_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/19997bff-55e7-42bd-bf6b-9372c89f7b5f_isrForCrosstalkSources_1125913_29.stderr parsl.dataflow.dflow DEBUG: Task 2081 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2081 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2081 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2081 has memoization hash 9e1c007716736ba3afe171161029823b parsl.dataflow.memoization INFO: Task 2081 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398196c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2081 try 0 launched on executor multi with executor id 2082 parsl.dataflow.dflow INFO: Standard output for task 2081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/ff0c05bc-e85d-4241-870b-37f834585f09_isrForCrosstalkSources_1125911_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/ff0c05bc-e85d-4241-870b-37f834585f09_isrForCrosstalkSources_1125911_39.stderr parsl.dataflow.dflow DEBUG: Task 2082 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2082 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2082 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2082 has memoization hash fc24f41574b3e74a09104d8b0e956c49 parsl.dataflow.memoization INFO: Task 2082 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2082 try 0 launched on executor multi with executor id 2083 parsl.dataflow.dflow INFO: Standard output for task 2082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/e0b67ef8-28d6-4c9e-a8c3-ffec87cc2c25_isrForCrosstalkSources_1125931_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/e0b67ef8-28d6-4c9e-a8c3-ffec87cc2c25_isrForCrosstalkSources_1125931_11.stderr parsl.dataflow.dflow DEBUG: Task 2083 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2083 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2083 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2083 has memoization hash 049d3ee7dc7ea8b30a0dfb3f60d7b7c1 parsl.dataflow.memoization INFO: Task 2083 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2083 try 0 launched on executor multi with executor id 2084 parsl.dataflow.dflow INFO: Standard output for task 2083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/aa2fbdad-8f0a-453b-b67b-e3da8b20e751_isrForCrosstalkSources_1125932_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/aa2fbdad-8f0a-453b-b67b-e3da8b20e751_isrForCrosstalkSources_1125932_16.stderr parsl.dataflow.dflow DEBUG: Task 2084 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2084 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2084 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2084 has memoization hash 3469f46cde1b954f11bec555bab13baf parsl.dataflow.memoization INFO: Task 2084 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398198a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2084 try 0 launched on executor multi with executor id 2085 parsl.dataflow.dflow INFO: Standard output for task 2084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/eb33f2a2-4fed-4ff6-98eb-116bb546b16b_isrForCrosstalkSources_1125892_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/eb33f2a2-4fed-4ff6-98eb-116bb546b16b_isrForCrosstalkSources_1125892_7.stderr parsl.dataflow.dflow DEBUG: Task 2085 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2085 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2085 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2085 has memoization hash c1af12ae482ced5e2fae4c48998f925d parsl.dataflow.memoization INFO: Task 2085 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2085 try 0 launched on executor multi with executor id 2086 parsl.dataflow.dflow INFO: Standard output for task 2085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/9d047b1b-e17a-4c44-b1d7-52e4d6874f0f_isrForCrosstalkSources_1125903_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/9d047b1b-e17a-4c44-b1d7-52e4d6874f0f_isrForCrosstalkSources_1125903_49.stderr parsl.dataflow.dflow DEBUG: Task 2086 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2086 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2086 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2086 has memoization hash 5bfee53454eaa798a67335ad09a09136 parsl.dataflow.memoization INFO: Task 2086 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398199e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2086 try 0 launched on executor multi with executor id 2087 parsl.dataflow.dflow INFO: Standard output for task 2086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/68fd3829-9a74-4dbb-b581-98d9eef7236a_isrForCrosstalkSources_1125896_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/68fd3829-9a74-4dbb-b581-98d9eef7236a_isrForCrosstalkSources_1125896_31.stderr parsl.dataflow.dflow DEBUG: Task 2087 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2087 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2087 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2087 has memoization hash 46985df8e47078d9d41e3cea3464ab4b parsl.dataflow.memoization INFO: Task 2087 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2087 try 0 launched on executor multi with executor id 2088 parsl.dataflow.dflow INFO: Standard output for task 2087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/0e68c4bc-9814-4bba-b03c-ee98d11d7389_isrForCrosstalkSources_1125889_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/0e68c4bc-9814-4bba-b03c-ee98d11d7389_isrForCrosstalkSources_1125889_4.stderr parsl.dataflow.dflow DEBUG: Task 2088 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2088 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2088 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2088 has memoization hash 6c09794193b0b922896c2c67fa0a65f1 parsl.dataflow.memoization INFO: Task 2088 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2088 try 0 launched on executor multi with executor id 2089 parsl.dataflow.dflow INFO: Standard output for task 2088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/22fd47ef-9fdf-415e-9e72-e829d6176666_isrForCrosstalkSources_1125940_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/22fd47ef-9fdf-415e-9e72-e829d6176666_isrForCrosstalkSources_1125940_41.stderr parsl.dataflow.dflow DEBUG: Task 2089 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2089 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2089 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2089 has memoization hash d866e31d48a102e7d10cb1e393df33a9 parsl.dataflow.memoization INFO: Task 2089 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2089 try 0 launched on executor multi with executor id 2090 parsl.dataflow.dflow INFO: Standard output for task 2089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/9e15afb7-f658-492b-b6de-8b4d489ca7dc_isrForCrosstalkSources_1125946_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/9e15afb7-f658-492b-b6de-8b4d489ca7dc_isrForCrosstalkSources_1125946_53.stderr parsl.dataflow.dflow DEBUG: Task 2090 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2090 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2090 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2090 has memoization hash 0259f4d2e2febad6bc8fe9f21fee3d23 parsl.dataflow.memoization INFO: Task 2090 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2090 try 0 launched on executor multi with executor id 2091 parsl.dataflow.dflow INFO: Standard output for task 2090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/abe69bc9-85a3-44ed-89b2-551135d909ca_isrForCrosstalkSources_1125889_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/abe69bc9-85a3-44ed-89b2-551135d909ca_isrForCrosstalkSources_1125889_45.stderr parsl.dataflow.dflow DEBUG: Task 2091 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2091 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2091 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2091 has memoization hash 936df403ed9032f66025da7945f08423 parsl.dataflow.memoization INFO: Task 2091 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2091 try 0 launched on executor multi with executor id 2092 parsl.dataflow.dflow INFO: Standard output for task 2091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/332b64bf-655f-4b32-8eaa-9d1e2548b7c0_isrForCrosstalkSources_1125896_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/332b64bf-655f-4b32-8eaa-9d1e2548b7c0_isrForCrosstalkSources_1125896_48.stderr parsl.dataflow.dflow DEBUG: Task 2092 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2092 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2092 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2092 has memoization hash d301df70951f8be56abd9253b22e9a5a parsl.dataflow.memoization INFO: Task 2092 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2092 try 0 launched on executor multi with executor id 2093 parsl.dataflow.dflow INFO: Standard output for task 2092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/888de858-75b8-4039-ab20-d9ce03792f79_isrForCrosstalkSources_1125890_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/888de858-75b8-4039-ab20-d9ce03792f79_isrForCrosstalkSources_1125890_11.stderr parsl.dataflow.dflow DEBUG: Task 2093 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2093 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2093 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2093 has memoization hash 4c08ee2040e47adf6852de726975bd53 parsl.dataflow.memoization INFO: Task 2093 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2093 try 0 launched on executor multi with executor id 2094 parsl.dataflow.dflow INFO: Standard output for task 2093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/138ebef2-cc56-453d-8d2e-017a63c12778_isrForCrosstalkSources_1125943_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/138ebef2-cc56-453d-8d2e-017a63c12778_isrForCrosstalkSources_1125943_60.stderr parsl.dataflow.dflow DEBUG: Task 2094 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2094 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2094 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2094 has memoization hash 8b09d308f2e3832809ad1387a3ae54fd parsl.dataflow.memoization INFO: Task 2094 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2094 try 0 launched on executor multi with executor id 2095 parsl.dataflow.dflow INFO: Standard output for task 2094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/ee448847-32a4-4559-9529-9a7ab02de663_isrForCrosstalkSources_1125928_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/ee448847-32a4-4559-9529-9a7ab02de663_isrForCrosstalkSources_1125928_61.stderr parsl.dataflow.dflow DEBUG: Task 2095 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2095 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2095 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2095 has memoization hash d392aa2391d5a9f7f14e78ec2c0a586c parsl.dataflow.memoization INFO: Task 2095 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39819f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2095 try 0 launched on executor multi with executor id 2096 parsl.dataflow.dflow INFO: Standard output for task 2095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/83fa7603-3741-4e2c-ac84-490193a63fe9_isrForCrosstalkSources_1125947_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/83fa7603-3741-4e2c-ac84-490193a63fe9_isrForCrosstalkSources_1125947_62.stderr parsl.dataflow.dflow DEBUG: Task 2096 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2096 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2096 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2096 has memoization hash a2386ef8a8e70fdaf0f00f84ddc8e6fe parsl.dataflow.memoization INFO: Task 2096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2096 try 0 launched on executor multi with executor id 2097 parsl.dataflow.dflow INFO: Standard output for task 2096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/e5091d49-0dd9-4d88-bdb0-2cc20a5d64f1_isrForCrosstalkSources_1125931_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/e5091d49-0dd9-4d88-bdb0-2cc20a5d64f1_isrForCrosstalkSources_1125931_20.stderr parsl.dataflow.dflow DEBUG: Task 2097 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2097 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2097 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2097 has memoization hash c240a42130ca532dbbfca380c66dbdc5 parsl.dataflow.memoization INFO: Task 2097 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2097 try 0 launched on executor multi with executor id 2098 parsl.dataflow.dflow INFO: Standard output for task 2097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/b088c8b8-7d5d-42ee-b575-18360994d9cf_isrForCrosstalkSources_1125940_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/b088c8b8-7d5d-42ee-b575-18360994d9cf_isrForCrosstalkSources_1125940_43.stderr parsl.dataflow.dflow DEBUG: Task 2098 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2098 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2098 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2098 has memoization hash ed5bda66030db3df928e8c8cfbdc27cc parsl.dataflow.memoization INFO: Task 2098 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2098 try 0 launched on executor multi with executor id 2099 parsl.dataflow.dflow INFO: Standard output for task 2098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/498b6140-615f-4049-a389-c0e979304f13_isrForCrosstalkSources_1125907_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/498b6140-615f-4049-a389-c0e979304f13_isrForCrosstalkSources_1125907_56.stderr parsl.dataflow.dflow DEBUG: Task 2099 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2099 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2099 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2099 has memoization hash 81eba4f68601a1cee48496df2214b001 parsl.dataflow.memoization INFO: Task 2099 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2099 try 0 launched on executor multi with executor id 2100 parsl.dataflow.dflow INFO: Standard output for task 2099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/71d24a3b-8f7c-4015-9c31-8e153dac30aa_isrForCrosstalkSources_1125925_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/71d24a3b-8f7c-4015-9c31-8e153dac30aa_isrForCrosstalkSources_1125925_31.stderr parsl.dataflow.dflow DEBUG: Task 2100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2100 has memoization hash f3c7bd77b3ef5b6305bbf6e0c9af7cff parsl.dataflow.memoization INFO: Task 2100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2100 try 0 launched on executor multi with executor id 2101 parsl.dataflow.dflow INFO: Standard output for task 2100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/064cff44-050d-4f0e-9dd8-2b46b56f58fe_isrForCrosstalkSources_1125912_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/064cff44-050d-4f0e-9dd8-2b46b56f58fe_isrForCrosstalkSources_1125912_50.stderr parsl.dataflow.dflow DEBUG: Task 2101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2101 has memoization hash 2ae32254a9db4e7e035576c378023d6b parsl.dataflow.memoization INFO: Task 2101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2101 try 0 launched on executor multi with executor id 2102 parsl.dataflow.dflow INFO: Standard output for task 2101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/dc37727c-668a-48a2-9b25-3eeb3b714c31_isrForCrosstalkSources_1125927_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/dc37727c-668a-48a2-9b25-3eeb3b714c31_isrForCrosstalkSources_1125927_35.stderr parsl.dataflow.dflow DEBUG: Task 2102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2102 has memoization hash 4448498643a4cf067c851c6b43c66bd1 parsl.dataflow.memoization INFO: Task 2102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2102 try 0 launched on executor multi with executor id 2103 parsl.dataflow.dflow INFO: Standard output for task 2102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/09f62f20-3fc9-45a9-a2dc-85b74400442e_isrForCrosstalkSources_1125906_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/09f62f20-3fc9-45a9-a2dc-85b74400442e_isrForCrosstalkSources_1125906_8.stderr parsl.dataflow.dflow DEBUG: Task 2103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2103 has memoization hash 6cc4673875d884a1b47d0e5d86f98207 parsl.dataflow.memoization INFO: Task 2103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2103 try 0 launched on executor multi with executor id 2104 parsl.dataflow.dflow INFO: Standard output for task 2103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f6533b11-8eac-4829-b8d0-f8c5a554aff1_isrForCrosstalkSources_1125945_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f6533b11-8eac-4829-b8d0-f8c5a554aff1_isrForCrosstalkSources_1125945_51.stderr parsl.dataflow.dflow DEBUG: Task 2104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2104 has memoization hash 1271c627ff81c9997d09e1fc00e54e9c parsl.dataflow.memoization INFO: Task 2104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2104 try 0 launched on executor multi with executor id 2105 parsl.dataflow.dflow INFO: Standard output for task 2104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/7c4dbb21-a9e7-4208-a4c7-132ca730e11b_isrForCrosstalkSources_1125888_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/7c4dbb21-a9e7-4208-a4c7-132ca730e11b_isrForCrosstalkSources_1125888_14.stderr parsl.dataflow.dflow DEBUG: Task 2105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2105 has memoization hash f70862fa386ab16f64de84a273c25a2e parsl.dataflow.memoization INFO: Task 2105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2105 try 0 launched on executor multi with executor id 2106 parsl.dataflow.dflow INFO: Standard output for task 2105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/aad7c8d3-1618-41be-b865-46bc4e73a360_isrForCrosstalkSources_1125931_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/aad7c8d3-1618-41be-b865-46bc4e73a360_isrForCrosstalkSources_1125931_22.stderr parsl.dataflow.dflow DEBUG: Task 2106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2106 has memoization hash 9bb9a09572f0d084245a700b4e92ac9c parsl.dataflow.memoization INFO: Task 2106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2106 try 0 launched on executor multi with executor id 2107 parsl.dataflow.dflow INFO: Standard output for task 2106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ed1724cf-d5b5-4f78-a656-f31fe3c4d57d_isrForCrosstalkSources_1125947_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ed1724cf-d5b5-4f78-a656-f31fe3c4d57d_isrForCrosstalkSources_1125947_40.stderr parsl.dataflow.dflow DEBUG: Task 2107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2107 has memoization hash 40b0154f64cef0ce535c45c0831cdad6 parsl.dataflow.memoization INFO: Task 2107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2107 try 0 launched on executor multi with executor id 2108 parsl.dataflow.dflow INFO: Standard output for task 2107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/20b9da95-089f-47fe-b64d-51be3f305188_isrForCrosstalkSources_1125942_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/20b9da95-089f-47fe-b64d-51be3f305188_isrForCrosstalkSources_1125942_40.stderr parsl.dataflow.dflow DEBUG: Task 2108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2108 has memoization hash 9fedb92454319bebdda70a10934609ec parsl.dataflow.memoization INFO: Task 2108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2108 try 0 launched on executor multi with executor id 2109 parsl.dataflow.dflow INFO: Standard output for task 2108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/5d8123f3-e5aa-4d24-9994-c661a433884c_isrForCrosstalkSources_1125898_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/5d8123f3-e5aa-4d24-9994-c661a433884c_isrForCrosstalkSources_1125898_50.stderr parsl.dataflow.dflow DEBUG: Task 2109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2109 has memoization hash a3303a0d1c77ddf319a24f6b1033caaf parsl.dataflow.memoization INFO: Task 2109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2109 try 0 launched on executor multi with executor id 2110 parsl.dataflow.dflow INFO: Standard output for task 2109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/119c123a-b786-4568-b22b-8a034403c63f_isrForCrosstalkSources_1125903_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/119c123a-b786-4568-b22b-8a034403c63f_isrForCrosstalkSources_1125903_17.stderr parsl.dataflow.dflow DEBUG: Task 2110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2110 has memoization hash e93a7c90f70b51e471da4d2020a8bf73 parsl.dataflow.memoization INFO: Task 2110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2110 try 0 launched on executor multi with executor id 2111 parsl.dataflow.dflow INFO: Standard output for task 2110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/49792d21-c2fe-47fa-a401-30623a09f1a2_isrForCrosstalkSources_1125941_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/49792d21-c2fe-47fa-a401-30623a09f1a2_isrForCrosstalkSources_1125941_40.stderr parsl.dataflow.dflow DEBUG: Task 2111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2111 has memoization hash c676703f697e3b6333436ff1555d59ee parsl.dataflow.memoization INFO: Task 2111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2111 try 0 launched on executor multi with executor id 2112 parsl.dataflow.dflow INFO: Standard output for task 2111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/988cf483-a1db-4948-84ec-b23b2bdda722_isrForCrosstalkSources_1125947_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/988cf483-a1db-4948-84ec-b23b2bdda722_isrForCrosstalkSources_1125947_59.stderr parsl.dataflow.dflow DEBUG: Task 2112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2112 has memoization hash b36a7d35dfafc30f952389062d479cc9 parsl.dataflow.memoization INFO: Task 2112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2112 try 0 launched on executor multi with executor id 2113 parsl.dataflow.dflow INFO: Standard output for task 2112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/dea42058-860c-47fe-9c0c-fe30c08e1164_isrForCrosstalkSources_1125926_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/dea42058-860c-47fe-9c0c-fe30c08e1164_isrForCrosstalkSources_1125926_43.stderr parsl.dataflow.dflow DEBUG: Task 2113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2113 has memoization hash 4b5f465879b82cad4b35ad6e6b98cb05 parsl.dataflow.memoization INFO: Task 2113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2113 try 0 launched on executor multi with executor id 2114 parsl.dataflow.dflow INFO: Standard output for task 2113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/4130db13-faab-4b8e-af99-1919fcddf50e_isrForCrosstalkSources_1125934_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/4130db13-faab-4b8e-af99-1919fcddf50e_isrForCrosstalkSources_1125934_60.stderr parsl.dataflow.dflow DEBUG: Task 2114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2114 has memoization hash 34fcd22cd7cd6e9899f9af19701296ab parsl.dataflow.memoization INFO: Task 2114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2114 try 0 launched on executor multi with executor id 2115 parsl.dataflow.dflow INFO: Standard output for task 2114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/35d0c911-c224-42de-8df2-c5bb7a8efa94_isrForCrosstalkSources_1125907_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/35d0c911-c224-42de-8df2-c5bb7a8efa94_isrForCrosstalkSources_1125907_49.stderr parsl.dataflow.dflow DEBUG: Task 2115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2115 has memoization hash 0d764e763083304cab2394b9becc8e07 parsl.dataflow.memoization INFO: Task 2115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2115 try 0 launched on executor multi with executor id 2116 parsl.dataflow.dflow INFO: Standard output for task 2115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/6b711895-faba-470e-b282-3786b9151907_isrForCrosstalkSources_1125937_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/6b711895-faba-470e-b282-3786b9151907_isrForCrosstalkSources_1125937_42.stderr parsl.dataflow.dflow DEBUG: Task 2116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2116 has memoization hash 068d5ca7bebbde2c8441ea9686865f87 parsl.dataflow.memoization INFO: Task 2116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2116 try 0 launched on executor multi with executor id 2117 parsl.dataflow.dflow INFO: Standard output for task 2116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/349fc28a-a905-4b6a-b665-3a8517869639_isrForCrosstalkSources_1125938_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/349fc28a-a905-4b6a-b665-3a8517869639_isrForCrosstalkSources_1125938_42.stderr parsl.dataflow.dflow DEBUG: Task 2117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2117 has memoization hash 42be4791077e58bcd1e3e3299e01a9e1 parsl.dataflow.memoization INFO: Task 2117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2117 try 0 launched on executor multi with executor id 2118 parsl.dataflow.dflow INFO: Standard output for task 2117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/e9b8d6b4-2476-4fb6-9c54-e2e81c5db3b0_isrForCrosstalkSources_1125915_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/e9b8d6b4-2476-4fb6-9c54-e2e81c5db3b0_isrForCrosstalkSources_1125915_55.stderr parsl.dataflow.dflow DEBUG: Task 2118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2118 has memoization hash 6e61896c443451425d679214b4d9515e parsl.dataflow.memoization INFO: Task 2118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2118 try 0 launched on executor multi with executor id 2119 parsl.dataflow.dflow INFO: Standard output for task 2118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/aa6247a2-0a8e-4a65-a608-11079638be48_isrForCrosstalkSources_1125902_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/aa6247a2-0a8e-4a65-a608-11079638be48_isrForCrosstalkSources_1125902_44.stderr parsl.dataflow.dflow DEBUG: Task 2119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2119 has memoization hash 15cc913976d2e6697b122ec4aade325e parsl.dataflow.memoization INFO: Task 2119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2119 try 0 launched on executor multi with executor id 2120 parsl.dataflow.dflow INFO: Standard output for task 2119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/9dde083b-fd0a-418d-a432-2b5b3e5a650f_isrForCrosstalkSources_1125916_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/9dde083b-fd0a-418d-a432-2b5b3e5a650f_isrForCrosstalkSources_1125916_42.stderr parsl.dataflow.dflow DEBUG: Task 2120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2120 has memoization hash 06ae4099921a9aa89612928d47d6e435 parsl.dataflow.memoization INFO: Task 2120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2120 try 0 launched on executor multi with executor id 2121 parsl.dataflow.dflow INFO: Standard output for task 2120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/8b7d6ae0-825a-4615-9716-9fc04bf860d6_isrForCrosstalkSources_1125906_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/8b7d6ae0-825a-4615-9716-9fc04bf860d6_isrForCrosstalkSources_1125906_10.stderr parsl.dataflow.dflow DEBUG: Task 2121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2121 has memoization hash 365c41311ba72f88ea3a69c9846ad62c parsl.dataflow.memoization INFO: Task 2121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2121 try 0 launched on executor multi with executor id 2122 parsl.dataflow.dflow INFO: Standard output for task 2121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/11d9dfd2-8aa0-4a4c-bcdd-e6a8e1957f41_isrForCrosstalkSources_1125910_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/11d9dfd2-8aa0-4a4c-bcdd-e6a8e1957f41_isrForCrosstalkSources_1125910_52.stderr parsl.dataflow.dflow DEBUG: Task 2122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2122 has memoization hash cd608375af68913f8bdb1c8100ec9146 parsl.dataflow.memoization INFO: Task 2122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2122 try 0 launched on executor multi with executor id 2123 parsl.dataflow.dflow INFO: Standard output for task 2122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/536c6d06-e71c-4a0b-aaf0-bb5071c7e13d_isrForCrosstalkSources_1125904_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/536c6d06-e71c-4a0b-aaf0-bb5071c7e13d_isrForCrosstalkSources_1125904_38.stderr parsl.dataflow.dflow DEBUG: Task 2123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2123 has memoization hash 2fd4cdff7bcd2de32d5d9705af34d417 parsl.dataflow.memoization INFO: Task 2123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2123 try 0 launched on executor multi with executor id 2124 parsl.dataflow.dflow INFO: Standard output for task 2123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/90a2fa78-6694-483f-94e4-794f33a12f91_isrForCrosstalkSources_1125945_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/90a2fa78-6694-483f-94e4-794f33a12f91_isrForCrosstalkSources_1125945_24.stderr parsl.dataflow.dflow DEBUG: Task 2124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2124 has memoization hash 06dfa003f904d9a7f98acac17477e485 parsl.dataflow.memoization INFO: Task 2124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2124 try 0 launched on executor multi with executor id 2125 parsl.dataflow.dflow INFO: Standard output for task 2124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/9a1ac523-c23c-4bba-98d3-be41cbbbd3b8_isrForCrosstalkSources_1125914_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/9a1ac523-c23c-4bba-98d3-be41cbbbd3b8_isrForCrosstalkSources_1125914_59.stderr parsl.dataflow.dflow DEBUG: Task 2125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2125 has memoization hash e01cc8f2c3c49f207679ce341718c686 parsl.dataflow.memoization INFO: Task 2125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2125 try 0 launched on executor multi with executor id 2126 parsl.dataflow.dflow INFO: Standard output for task 2125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/f74b16f7-7226-4668-8a85-e45dc717de34_isrForCrosstalkSources_1125903_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/f74b16f7-7226-4668-8a85-e45dc717de34_isrForCrosstalkSources_1125903_52.stderr parsl.dataflow.dflow DEBUG: Task 2126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2126 has memoization hash 3cc0360019a029951277135c02a1e7e6 parsl.dataflow.memoization INFO: Task 2126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2126 try 0 launched on executor multi with executor id 2127 parsl.dataflow.dflow INFO: Standard output for task 2126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/96f945a3-3dd9-45e4-9c29-79cbe4a6826d_isrForCrosstalkSources_1125895_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/96f945a3-3dd9-45e4-9c29-79cbe4a6826d_isrForCrosstalkSources_1125895_17.stderr parsl.dataflow.dflow DEBUG: Task 2127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2127 has memoization hash 223ed84c2fc9509fd7eb6e73b859d9f8 parsl.dataflow.memoization INFO: Task 2127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2127 try 0 launched on executor multi with executor id 2128 parsl.dataflow.dflow INFO: Standard output for task 2127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/ca58a9d5-9063-452f-8bb5-5669f6a2a809_isrForCrosstalkSources_1125909_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/ca58a9d5-9063-452f-8bb5-5669f6a2a809_isrForCrosstalkSources_1125909_10.stderr parsl.dataflow.dflow DEBUG: Task 2128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2128 has memoization hash 23ed24e48900fcff0af82a088d4f1491 parsl.dataflow.memoization INFO: Task 2128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2128 try 0 launched on executor multi with executor id 2129 parsl.dataflow.dflow INFO: Standard output for task 2128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/7b8556e0-72ce-4dfd-8b6d-0301746a61af_isrForCrosstalkSources_1125921_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/7b8556e0-72ce-4dfd-8b6d-0301746a61af_isrForCrosstalkSources_1125921_16.stderr parsl.dataflow.dflow DEBUG: Task 2129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2129 has memoization hash 135d2275dfbab1c31c5cf9282e083f46 parsl.dataflow.memoization INFO: Task 2129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2129 try 0 launched on executor multi with executor id 2130 parsl.dataflow.dflow INFO: Standard output for task 2129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/0dffbb12-26b9-4ffc-bad5-aac369baf1d2_isrForCrosstalkSources_1125925_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/0dffbb12-26b9-4ffc-bad5-aac369baf1d2_isrForCrosstalkSources_1125925_17.stderr parsl.dataflow.dflow DEBUG: Task 2130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2130 has memoization hash 72879d176cf17209febbeb6a2d8f6733 parsl.dataflow.memoization INFO: Task 2130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2130 try 0 launched on executor multi with executor id 2131 parsl.dataflow.dflow INFO: Standard output for task 2130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/7514619e-1e3d-47fe-a696-de63c81511aa_isrForCrosstalkSources_1125946_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/7514619e-1e3d-47fe-a696-de63c81511aa_isrForCrosstalkSources_1125946_27.stderr parsl.dataflow.dflow DEBUG: Task 2131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2131 has memoization hash 0cf9204a99f870424bd72c4d7c2aa0db parsl.dataflow.memoization INFO: Task 2131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2131 try 0 launched on executor multi with executor id 2132 parsl.dataflow.dflow INFO: Standard output for task 2131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/aad77c46-c611-4c17-9f14-f6bcf561b0d6_isrForCrosstalkSources_1125904_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/aad77c46-c611-4c17-9f14-f6bcf561b0d6_isrForCrosstalkSources_1125904_21.stderr parsl.dataflow.dflow DEBUG: Task 2132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2132 has memoization hash f0507f5a68f57528b4d2171c20914f93 parsl.dataflow.memoization INFO: Task 2132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2132 try 0 launched on executor multi with executor id 2133 parsl.dataflow.dflow INFO: Standard output for task 2132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/7ac27915-cd73-4553-8686-b1b7ec296705_isrForCrosstalkSources_1125925_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/7ac27915-cd73-4553-8686-b1b7ec296705_isrForCrosstalkSources_1125925_7.stderr parsl.dataflow.dflow DEBUG: Task 2133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2133 has memoization hash be243659f5a39b36984ba0932b3ac3fd parsl.dataflow.memoization INFO: Task 2133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2133 try 0 launched on executor multi with executor id 2134 parsl.dataflow.dflow INFO: Standard output for task 2133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ba6cce75-d597-4e91-8748-f47f9d5f5d05_isrForCrosstalkSources_1125918_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ba6cce75-d597-4e91-8748-f47f9d5f5d05_isrForCrosstalkSources_1125918_39.stderr parsl.dataflow.dflow DEBUG: Task 2134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2134 has memoization hash 2cca9ca02bccb3bbf35de6ebe8a60f91 parsl.dataflow.memoization INFO: Task 2134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2134 try 0 launched on executor multi with executor id 2135 parsl.dataflow.dflow INFO: Standard output for task 2134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/8ae05ca8-85d4-4c42-945a-6e7c8f78ef8c_isrForCrosstalkSources_1125922_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/8ae05ca8-85d4-4c42-945a-6e7c8f78ef8c_isrForCrosstalkSources_1125922_57.stderr parsl.dataflow.dflow DEBUG: Task 2135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2135 has memoization hash 0a1eedd9e6cfdc4343cf289a38b5872e parsl.dataflow.memoization INFO: Task 2135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2135 try 0 launched on executor multi with executor id 2136 parsl.dataflow.dflow INFO: Standard output for task 2135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/46b6c499-1fe2-4e50-90a1-547da70dff2f_isrForCrosstalkSources_1125906_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/46b6c499-1fe2-4e50-90a1-547da70dff2f_isrForCrosstalkSources_1125906_58.stderr parsl.dataflow.dflow DEBUG: Task 2136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2136 has memoization hash b0dc51e1fc982f7a151bd2345ea2883e parsl.dataflow.memoization INFO: Task 2136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2136 try 0 launched on executor multi with executor id 2137 parsl.dataflow.dflow INFO: Standard output for task 2136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/4897e0a1-5d20-4aab-a72b-56785ba2fc56_isrForCrosstalkSources_1125914_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/4897e0a1-5d20-4aab-a72b-56785ba2fc56_isrForCrosstalkSources_1125914_32.stderr parsl.dataflow.dflow DEBUG: Task 2137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2137 has memoization hash ffae66cd978c7dd0af4ad5a43bf023a6 parsl.dataflow.memoization INFO: Task 2137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2137 try 0 launched on executor multi with executor id 2138 parsl.dataflow.dflow INFO: Standard output for task 2137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/1f63d591-24a3-4181-90b8-f9ed92cb25b6_isrForCrosstalkSources_1125914_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/1f63d591-24a3-4181-90b8-f9ed92cb25b6_isrForCrosstalkSources_1125914_6.stderr parsl.dataflow.dflow DEBUG: Task 2138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2138 has memoization hash c3cd305c5406311317dfecb5fa1d8c67 parsl.dataflow.memoization INFO: Task 2138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2138 try 0 launched on executor multi with executor id 2139 parsl.dataflow.dflow INFO: Standard output for task 2138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/49f0a26b-293c-4aa4-ab1b-f5d0533e5c94_isrForCrosstalkSources_1125913_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/49f0a26b-293c-4aa4-ab1b-f5d0533e5c94_isrForCrosstalkSources_1125913_59.stderr parsl.dataflow.dflow DEBUG: Task 2139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2139 has memoization hash 6c1e73fc652074d24fcc0b710361421b parsl.dataflow.memoization INFO: Task 2139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2139 try 0 launched on executor multi with executor id 2140 parsl.dataflow.dflow INFO: Standard output for task 2139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/84a444d8-4663-454f-9b66-0b85f2a38731_isrForCrosstalkSources_1125928_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/84a444d8-4663-454f-9b66-0b85f2a38731_isrForCrosstalkSources_1125928_41.stderr parsl.dataflow.dflow DEBUG: Task 2140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2140 has memoization hash 9627c9bd802422c2623dc0d75d3992e3 parsl.dataflow.memoization INFO: Task 2140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2140 try 0 launched on executor multi with executor id 2141 parsl.dataflow.dflow INFO: Standard output for task 2140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/d8efcd71-3b76-43bb-bc98-ca905ecd33bb_isrForCrosstalkSources_1125928_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/d8efcd71-3b76-43bb-bc98-ca905ecd33bb_isrForCrosstalkSources_1125928_55.stderr parsl.dataflow.dflow DEBUG: Task 2141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2141 has memoization hash e8daab1f84e8e4b99375ae9db0432761 parsl.dataflow.memoization INFO: Task 2141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2141 try 0 launched on executor multi with executor id 2142 parsl.dataflow.dflow INFO: Standard output for task 2141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/8bccd0e1-98d6-48f8-9041-0a95fb40e9ea_isrForCrosstalkSources_1125922_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/8bccd0e1-98d6-48f8-9041-0a95fb40e9ea_isrForCrosstalkSources_1125922_33.stderr parsl.dataflow.dflow DEBUG: Task 2142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2142 has memoization hash 509410072045a01a44e4839838af388e parsl.dataflow.memoization INFO: Task 2142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2142 try 0 launched on executor multi with executor id 2143 parsl.dataflow.dflow INFO: Standard output for task 2142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/47031b68-2ba9-4644-92f5-2218b6df5f9c_isrForCrosstalkSources_1125941_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/47031b68-2ba9-4644-92f5-2218b6df5f9c_isrForCrosstalkSources_1125941_29.stderr parsl.dataflow.dflow DEBUG: Task 2143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2143 has memoization hash 2f5ce6118f6f4e1e9a52420281619533 parsl.dataflow.memoization INFO: Task 2143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2143 try 0 launched on executor multi with executor id 2144 parsl.dataflow.dflow INFO: Standard output for task 2143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/4ca6c849-7a97-490e-876c-52b1f901e94b_isrForCrosstalkSources_1125922_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/4ca6c849-7a97-490e-876c-52b1f901e94b_isrForCrosstalkSources_1125922_52.stderr parsl.dataflow.dflow DEBUG: Task 2144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2144 has memoization hash 00e923b770db972c1cfba70fc2c44852 parsl.dataflow.memoization INFO: Task 2144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2144 try 0 launched on executor multi with executor id 2145 parsl.dataflow.dflow INFO: Standard output for task 2144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/7a2cc30f-b9e5-49da-bd8b-43f7063482dc_isrForCrosstalkSources_1125945_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/7a2cc30f-b9e5-49da-bd8b-43f7063482dc_isrForCrosstalkSources_1125945_58.stderr parsl.dataflow.dflow DEBUG: Task 2145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2145 has memoization hash add9d8f5da6dda60d1a877045009f662 parsl.dataflow.memoization INFO: Task 2145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2145 try 0 launched on executor multi with executor id 2146 parsl.dataflow.dflow INFO: Standard output for task 2145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/0e8230c4-1d49-4ecd-abcc-3bcbffb7bb61_isrForCrosstalkSources_1125942_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/0e8230c4-1d49-4ecd-abcc-3bcbffb7bb61_isrForCrosstalkSources_1125942_6.stderr parsl.dataflow.dflow DEBUG: Task 2146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2146 has memoization hash ced3be8610897cbd77fd1bc65fc59cdb parsl.dataflow.memoization INFO: Task 2146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3981bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2146 try 0 launched on executor multi with executor id 2147 parsl.dataflow.dflow INFO: Standard output for task 2146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/15086f69-5bb4-4cbc-b706-1a3b44788bf0_isrForCrosstalkSources_1125916_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/15086f69-5bb4-4cbc-b706-1a3b44788bf0_isrForCrosstalkSources_1125916_5.stderr parsl.dataflow.dflow DEBUG: Task 2147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2147 has memoization hash 737a6afe8853d4bea092f6f2295bef3b parsl.dataflow.memoization INFO: Task 2147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39818040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2147 try 0 launched on executor multi with executor id 2148 parsl.dataflow.dflow INFO: Standard output for task 2147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/c6c291b5-b192-4d82-9852-d59f276c8596_isrForCrosstalkSources_1125937_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/c6c291b5-b192-4d82-9852-d59f276c8596_isrForCrosstalkSources_1125937_48.stderr parsl.dataflow.dflow DEBUG: Task 2148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2148 has memoization hash f7a22822c09a0207a0137b9f5ed51db5 parsl.dataflow.memoization INFO: Task 2148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2148 try 0 launched on executor multi with executor id 2149 parsl.dataflow.dflow INFO: Standard output for task 2148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/ba81997b-23d1-4e90-8f12-4aa892d33d2a_isrForCrosstalkSources_1125921_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/ba81997b-23d1-4e90-8f12-4aa892d33d2a_isrForCrosstalkSources_1125921_24.stderr parsl.dataflow.dflow DEBUG: Task 2149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2149 has memoization hash 9327f3a6b8cb4769f94de0fecd525fb6 parsl.dataflow.memoization INFO: Task 2149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2149 try 0 launched on executor multi with executor id 2150 parsl.dataflow.dflow INFO: Standard output for task 2149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/4190716a-b3aa-41ab-993f-090bd052a472_isrForCrosstalkSources_1125892_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/4190716a-b3aa-41ab-993f-090bd052a472_isrForCrosstalkSources_1125892_41.stderr parsl.dataflow.dflow DEBUG: Task 2150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2150 has memoization hash 58c4466b28651413cc558932bac941b2 parsl.dataflow.memoization INFO: Task 2150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2150 try 0 launched on executor multi with executor id 2151 parsl.dataflow.dflow INFO: Standard output for task 2150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/1c3da744-13ec-4996-b823-e1a0c7ea4527_isrForCrosstalkSources_1125917_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/1c3da744-13ec-4996-b823-e1a0c7ea4527_isrForCrosstalkSources_1125917_46.stderr parsl.dataflow.dflow DEBUG: Task 2151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2151 has memoization hash 0d9449e3dc44142e384ae722456c3d51 parsl.dataflow.memoization INFO: Task 2151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2151 try 0 launched on executor multi with executor id 2152 parsl.dataflow.dflow INFO: Standard output for task 2151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/3caf0e6c-5069-444c-8fb6-3d46aa228f71_isrForCrosstalkSources_1125915_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/3caf0e6c-5069-444c-8fb6-3d46aa228f71_isrForCrosstalkSources_1125915_53.stderr parsl.dataflow.dflow DEBUG: Task 2152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2152 has memoization hash 39110028ebcb7a7a49c6ff69b38db373 parsl.dataflow.memoization INFO: Task 2152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2152 try 0 launched on executor multi with executor id 2153 parsl.dataflow.dflow INFO: Standard output for task 2152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/916ab6cf-3e79-4c5b-a833-4723159a2ee4_isrForCrosstalkSources_1125921_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/916ab6cf-3e79-4c5b-a833-4723159a2ee4_isrForCrosstalkSources_1125921_29.stderr parsl.dataflow.dflow DEBUG: Task 2153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2153 has memoization hash d3abc721a8c7aed5e29d411af1e372a2 parsl.dataflow.memoization INFO: Task 2153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2153 try 0 launched on executor multi with executor id 2154 parsl.dataflow.dflow INFO: Standard output for task 2153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/99fba34e-acc7-4ef1-9a50-77da97b1da33_isrForCrosstalkSources_1125931_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/99fba34e-acc7-4ef1-9a50-77da97b1da33_isrForCrosstalkSources_1125931_1.stderr parsl.dataflow.dflow DEBUG: Task 2154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2154 has memoization hash 8fe19d2f79bb8e2dac0ca0c3e9f0ee13 parsl.dataflow.memoization INFO: Task 2154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2154 try 0 launched on executor multi with executor id 2155 parsl.dataflow.dflow INFO: Standard output for task 2154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/a79d0588-6250-42cd-b798-085a96e8bb4f_isrForCrosstalkSources_1125895_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/a79d0588-6250-42cd-b798-085a96e8bb4f_isrForCrosstalkSources_1125895_30.stderr parsl.dataflow.dflow DEBUG: Task 2155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2155 has memoization hash 36a7144b94b18255abded858c71721ef parsl.dataflow.memoization INFO: Task 2155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2155 try 0 launched on executor multi with executor id 2156 parsl.dataflow.dflow INFO: Standard output for task 2155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/7dba3dd4-6f6e-4e7f-aa5c-3da54cb29dcc_isrForCrosstalkSources_1125932_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/7dba3dd4-6f6e-4e7f-aa5c-3da54cb29dcc_isrForCrosstalkSources_1125932_43.stderr parsl.dataflow.dflow DEBUG: Task 2156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2156 has memoization hash 53f660bbd0677747d6d3d5555b823606 parsl.dataflow.memoization INFO: Task 2156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2156 try 0 launched on executor multi with executor id 2157 parsl.dataflow.dflow INFO: Standard output for task 2156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/2676f32f-b2db-40cf-a312-3df0cac28b64_isrForCrosstalkSources_1125947_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/2676f32f-b2db-40cf-a312-3df0cac28b64_isrForCrosstalkSources_1125947_56.stderr parsl.dataflow.dflow DEBUG: Task 2157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2157 has memoization hash 85f1f067e5dba322190f8bf2cb65dcc4 parsl.dataflow.memoization INFO: Task 2157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2157 try 0 launched on executor multi with executor id 2158 parsl.dataflow.dflow INFO: Standard output for task 2157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f5f5c472-1872-49fb-aa90-53d0040c21d2_isrForCrosstalkSources_1125910_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f5f5c472-1872-49fb-aa90-53d0040c21d2_isrForCrosstalkSources_1125910_24.stderr parsl.dataflow.dflow DEBUG: Task 2158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2158 has memoization hash c847a7f6ecb7007e9b61ad76fe3ecb2f parsl.dataflow.memoization INFO: Task 2158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2158 try 0 launched on executor multi with executor id 2159 parsl.dataflow.dflow INFO: Standard output for task 2158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/e4719521-e0ec-4a77-9fac-d94865793df7_isrForCrosstalkSources_1125919_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/e4719521-e0ec-4a77-9fac-d94865793df7_isrForCrosstalkSources_1125919_16.stderr parsl.dataflow.dflow DEBUG: Task 2159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2159 has memoization hash d69403bf47e4057ceb05d68dddd7c8de parsl.dataflow.memoization INFO: Task 2159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2159 try 0 launched on executor multi with executor id 2160 parsl.dataflow.dflow INFO: Standard output for task 2159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/0e40d68a-b1e1-4f1c-8eb8-ad777fa9b90a_isrForCrosstalkSources_1125897_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/0e40d68a-b1e1-4f1c-8eb8-ad777fa9b90a_isrForCrosstalkSources_1125897_46.stderr parsl.dataflow.dflow DEBUG: Task 2160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2160 has memoization hash 66617b686903f11c97c5ef4ab2285380 parsl.dataflow.memoization INFO: Task 2160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2160 try 0 launched on executor multi with executor id 2161 parsl.dataflow.dflow INFO: Standard output for task 2160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/9a3173d5-5b06-45c2-9f00-35ad8de7a3ba_isrForCrosstalkSources_1125943_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/9a3173d5-5b06-45c2-9f00-35ad8de7a3ba_isrForCrosstalkSources_1125943_33.stderr parsl.dataflow.dflow DEBUG: Task 2161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2161 has memoization hash 46597bdcf78bf47e8c79e4abad3acfa1 parsl.dataflow.memoization INFO: Task 2161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2161 try 0 launched on executor multi with executor id 2162 parsl.dataflow.dflow INFO: Standard output for task 2161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/420767d4-810c-420a-a189-dc37b1096196_isrForCrosstalkSources_1125909_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/420767d4-810c-420a-a189-dc37b1096196_isrForCrosstalkSources_1125909_41.stderr parsl.dataflow.dflow DEBUG: Task 2162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2162 has memoization hash 834bf622ae8efb89755e7d77acb99f7c parsl.dataflow.memoization INFO: Task 2162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2162 try 0 launched on executor multi with executor id 2163 parsl.dataflow.dflow INFO: Standard output for task 2162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/693ebe42-ed5f-4c78-b5e4-42d509d2ee1e_isrForCrosstalkSources_1125918_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/693ebe42-ed5f-4c78-b5e4-42d509d2ee1e_isrForCrosstalkSources_1125918_19.stderr parsl.dataflow.dflow DEBUG: Task 2163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2163 has memoization hash abff6a2086e8b3a2fde6b3b42fced00d parsl.dataflow.memoization INFO: Task 2163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2163 try 0 launched on executor multi with executor id 2164 parsl.dataflow.dflow INFO: Standard output for task 2163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/2e049568-a9c5-4b12-8406-0f6a302bb964_isrForCrosstalkSources_1125929_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/2e049568-a9c5-4b12-8406-0f6a302bb964_isrForCrosstalkSources_1125929_60.stderr parsl.dataflow.dflow DEBUG: Task 2164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2164 has memoization hash 919378ab5094a14acc3e19fc001d1a0d parsl.dataflow.memoization INFO: Task 2164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2164 try 0 launched on executor multi with executor id 2165 parsl.dataflow.dflow INFO: Standard output for task 2164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/65aeabb2-e8ca-4362-975a-1f0a09a516d7_isrForCrosstalkSources_1125941_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/65aeabb2-e8ca-4362-975a-1f0a09a516d7_isrForCrosstalkSources_1125941_8.stderr parsl.dataflow.dflow DEBUG: Task 2165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2165 has memoization hash 9a9b9175e0b1d9cf68dd05cc4f7b4056 parsl.dataflow.memoization INFO: Task 2165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2165 try 0 launched on executor multi with executor id 2166 parsl.dataflow.dflow INFO: Standard output for task 2165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4feee85d-5800-4c4a-b649-31fc10198f9d_isrForCrosstalkSources_1125908_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4feee85d-5800-4c4a-b649-31fc10198f9d_isrForCrosstalkSources_1125908_55.stderr parsl.dataflow.dflow DEBUG: Task 2166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2166 has memoization hash 4305b57940a528ff244b057ed682b1a8 parsl.dataflow.memoization INFO: Task 2166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2166 try 0 launched on executor multi with executor id 2167 parsl.dataflow.dflow INFO: Standard output for task 2166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/a1de2e30-9371-4955-b9a3-90147d300734_isrForCrosstalkSources_1125914_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/a1de2e30-9371-4955-b9a3-90147d300734_isrForCrosstalkSources_1125914_1.stderr parsl.dataflow.dflow DEBUG: Task 2167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2167 has memoization hash ab51807121218a4ebd9c0b75487f6b7a parsl.dataflow.memoization INFO: Task 2167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2167 try 0 launched on executor multi with executor id 2168 parsl.dataflow.dflow INFO: Standard output for task 2167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/cc3fe9bb-98ac-4af5-850f-ffa95d38ba2e_isrForCrosstalkSources_1125891_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/cc3fe9bb-98ac-4af5-850f-ffa95d38ba2e_isrForCrosstalkSources_1125891_60.stderr parsl.dataflow.dflow DEBUG: Task 2168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2168 has memoization hash 28c3669abd53d530e95f009d04dc7736 parsl.dataflow.memoization INFO: Task 2168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2168 try 0 launched on executor multi with executor id 2169 parsl.dataflow.dflow INFO: Standard output for task 2168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/cb76ebca-2b16-4ed6-9ad2-602568af5b31_isrForCrosstalkSources_1125909_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/cb76ebca-2b16-4ed6-9ad2-602568af5b31_isrForCrosstalkSources_1125909_62.stderr parsl.dataflow.dflow DEBUG: Task 2169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2169 has memoization hash b6b49850f5e5bd6fb1c9cd291dbfb5a5 parsl.dataflow.memoization INFO: Task 2169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2169 try 0 launched on executor multi with executor id 2170 parsl.dataflow.dflow INFO: Standard output for task 2169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/31e68a55-2db6-4cb1-bbb0-4210ce048c7f_isrForCrosstalkSources_1125917_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/31e68a55-2db6-4cb1-bbb0-4210ce048c7f_isrForCrosstalkSources_1125917_62.stderr parsl.dataflow.dflow DEBUG: Task 2170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2170 has memoization hash 449a4e1de33b1342aa251578ab3f9ac9 parsl.dataflow.memoization INFO: Task 2170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2170 try 0 launched on executor multi with executor id 2171 parsl.dataflow.dflow INFO: Standard output for task 2170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/784ee28b-ad64-4bac-a7c5-5494126b7498_isrForCrosstalkSources_1125888_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/784ee28b-ad64-4bac-a7c5-5494126b7498_isrForCrosstalkSources_1125888_18.stderr parsl.dataflow.dflow DEBUG: Task 2171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2171 has memoization hash f26480bd4be6f72a572c4d8211d97673 parsl.dataflow.memoization INFO: Task 2171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2171 try 0 launched on executor multi with executor id 2172 parsl.dataflow.dflow INFO: Standard output for task 2171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/f6fe3b42-1f94-4f06-9267-7964a7a6b9a9_isrForCrosstalkSources_1125933_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/f6fe3b42-1f94-4f06-9267-7964a7a6b9a9_isrForCrosstalkSources_1125933_20.stderr parsl.dataflow.dflow DEBUG: Task 2172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2172 has memoization hash e6d579056df14c4709131dbc60fa9deb parsl.dataflow.memoization INFO: Task 2172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2172 try 0 launched on executor multi with executor id 2173 parsl.dataflow.dflow INFO: Standard output for task 2172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c2dad2df-778b-4337-b4ec-954a7030f830_isrForCrosstalkSources_1125917_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c2dad2df-778b-4337-b4ec-954a7030f830_isrForCrosstalkSources_1125917_30.stderr parsl.dataflow.dflow DEBUG: Task 2173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2173 has memoization hash 0ab0f42fdc58404f2dcbe0dcc13b0656 parsl.dataflow.memoization INFO: Task 2173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2173 try 0 launched on executor multi with executor id 2174 parsl.dataflow.dflow INFO: Standard output for task 2173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/cb480439-3576-426c-ab23-73bf1b5c7e8b_isrForCrosstalkSources_1125928_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/cb480439-3576-426c-ab23-73bf1b5c7e8b_isrForCrosstalkSources_1125928_10.stderr parsl.dataflow.dflow DEBUG: Task 2174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2174 has memoization hash c776044168adbfa1b41dc44ddd3b8381 parsl.dataflow.memoization INFO: Task 2174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2174 try 0 launched on executor multi with executor id 2175 parsl.dataflow.dflow INFO: Standard output for task 2174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/6c2e1c86-bb1d-484f-9c98-7e58355bef93_isrForCrosstalkSources_1125928_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/6c2e1c86-bb1d-484f-9c98-7e58355bef93_isrForCrosstalkSources_1125928_24.stderr parsl.dataflow.dflow DEBUG: Task 2175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2175 has memoization hash 9cc28936462fa4b5871509ad408254fc parsl.dataflow.memoization INFO: Task 2175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2175 try 0 launched on executor multi with executor id 2176 parsl.dataflow.dflow INFO: Standard output for task 2175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/44963849-d5bf-4350-9f52-7fdf6121779b_isrForCrosstalkSources_1125922_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/44963849-d5bf-4350-9f52-7fdf6121779b_isrForCrosstalkSources_1125922_56.stderr parsl.dataflow.dflow DEBUG: Task 2176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2176 has memoization hash 8137e4015c5b0dcd3985e3a31ca136f0 parsl.dataflow.memoization INFO: Task 2176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2176 try 0 launched on executor multi with executor id 2177 parsl.dataflow.dflow INFO: Standard output for task 2176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/29cb678b-5127-4e34-87f5-3a797992e083_isrForCrosstalkSources_1125899_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/29cb678b-5127-4e34-87f5-3a797992e083_isrForCrosstalkSources_1125899_58.stderr parsl.dataflow.dflow DEBUG: Task 2177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2177 has memoization hash 20fc1aaa270d9372a69860377aa8f1f5 parsl.dataflow.memoization INFO: Task 2177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2177 try 0 launched on executor multi with executor id 2178 parsl.dataflow.dflow INFO: Standard output for task 2177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/ad19ac51-23dc-4414-942d-71032bd71166_isrForCrosstalkSources_1125898_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/ad19ac51-23dc-4414-942d-71032bd71166_isrForCrosstalkSources_1125898_4.stderr parsl.dataflow.dflow DEBUG: Task 2178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2178 has memoization hash 24316e3e8e8e73af32b39c4b137d2c37 parsl.dataflow.memoization INFO: Task 2178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2178 try 0 launched on executor multi with executor id 2179 parsl.dataflow.dflow INFO: Standard output for task 2178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/d528b682-c3a5-4492-86f7-701eadcf4d6d_isrForCrosstalkSources_1125903_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/d528b682-c3a5-4492-86f7-701eadcf4d6d_isrForCrosstalkSources_1125903_44.stderr parsl.dataflow.dflow DEBUG: Task 2179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2179 has memoization hash ec1dbec8d038ec40316d9d0bded8429b parsl.dataflow.memoization INFO: Task 2179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2179 try 0 launched on executor multi with executor id 2180 parsl.dataflow.dflow INFO: Standard output for task 2179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/347f5431-a5c7-4c2d-a0e5-42bd343950bb_isrForCrosstalkSources_1125893_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/347f5431-a5c7-4c2d-a0e5-42bd343950bb_isrForCrosstalkSources_1125893_22.stderr parsl.dataflow.dflow DEBUG: Task 2180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2180 has memoization hash df30189c5e9676a0e6601a6589f949be parsl.dataflow.memoization INFO: Task 2180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2180 try 0 launched on executor multi with executor id 2181 parsl.dataflow.dflow INFO: Standard output for task 2180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/6224a97a-8575-4ac9-abd5-4e7be9a6dbd3_isrForCrosstalkSources_1125911_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/6224a97a-8575-4ac9-abd5-4e7be9a6dbd3_isrForCrosstalkSources_1125911_4.stderr parsl.dataflow.dflow DEBUG: Task 2181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2181 has memoization hash ad3d2b70246fcc4eb164df6ef123142b parsl.dataflow.memoization INFO: Task 2181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2181 try 0 launched on executor multi with executor id 2182 parsl.dataflow.dflow INFO: Standard output for task 2181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/0bae3cce-88ed-47f5-b566-1ff8240ef289_isrForCrosstalkSources_1125917_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/0bae3cce-88ed-47f5-b566-1ff8240ef289_isrForCrosstalkSources_1125917_17.stderr parsl.dataflow.dflow DEBUG: Task 2182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2182 has memoization hash 0a8e20399a6f606acaa8cf5a8a8d3cce parsl.dataflow.memoization INFO: Task 2182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2182 try 0 launched on executor multi with executor id 2183 parsl.dataflow.dflow INFO: Standard output for task 2182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/a61c75bb-7af4-4ed7-800d-f955251fbb58_isrForCrosstalkSources_1125947_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/a61c75bb-7af4-4ed7-800d-f955251fbb58_isrForCrosstalkSources_1125947_35.stderr parsl.dataflow.dflow DEBUG: Task 2183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2183 has memoization hash e5d1cc3700edff885974f67d7f262929 parsl.dataflow.memoization INFO: Task 2183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2183 try 0 launched on executor multi with executor id 2184 parsl.dataflow.dflow INFO: Standard output for task 2183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/28c8c9ce-113a-4d41-a2ea-7a5eed38e828_isrForCrosstalkSources_1125936_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/28c8c9ce-113a-4d41-a2ea-7a5eed38e828_isrForCrosstalkSources_1125936_54.stderr parsl.dataflow.dflow DEBUG: Task 2184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2184 has memoization hash 8c9988819b681dee9b5ee07eb2e1ca34 parsl.dataflow.memoization INFO: Task 2184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2184 try 0 launched on executor multi with executor id 2185 parsl.dataflow.dflow INFO: Standard output for task 2184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/b8235e23-faf7-4d05-bb01-85cb683d69e7_isrForCrosstalkSources_1125918_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/b8235e23-faf7-4d05-bb01-85cb683d69e7_isrForCrosstalkSources_1125918_55.stderr parsl.dataflow.dflow DEBUG: Task 2185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2185 has memoization hash 7fb439c6e1030760a343574360e782f2 parsl.dataflow.memoization INFO: Task 2185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2185 try 0 launched on executor multi with executor id 2186 parsl.dataflow.dflow INFO: Standard output for task 2185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/c6b86485-5380-4e2c-93c4-5042d52d40ae_isrForCrosstalkSources_1125918_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/c6b86485-5380-4e2c-93c4-5042d52d40ae_isrForCrosstalkSources_1125918_43.stderr parsl.dataflow.dflow DEBUG: Task 2186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2186 has memoization hash 8842813924cb873c97e71f168afe0115 parsl.dataflow.memoization INFO: Task 2186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2186 try 0 launched on executor multi with executor id 2187 parsl.dataflow.dflow INFO: Standard output for task 2186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/9ee939d3-66d9-4a85-b3e6-e61534285df0_isrForCrosstalkSources_1125939_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/9ee939d3-66d9-4a85-b3e6-e61534285df0_isrForCrosstalkSources_1125939_13.stderr parsl.dataflow.dflow DEBUG: Task 2187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2187 has memoization hash d2768c137c2ddae25ce7a46976a0d153 parsl.dataflow.memoization INFO: Task 2187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2187 try 0 launched on executor multi with executor id 2188 parsl.dataflow.dflow INFO: Standard output for task 2187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/0cad6096-be2a-4f4f-af27-2e97573cf7c7_isrForCrosstalkSources_1125900_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/0cad6096-be2a-4f4f-af27-2e97573cf7c7_isrForCrosstalkSources_1125900_6.stderr parsl.dataflow.dflow DEBUG: Task 2188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2188 has memoization hash 523ba4bba64264a1cacaa207573d1ba3 parsl.dataflow.memoization INFO: Task 2188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2188 try 0 launched on executor multi with executor id 2189 parsl.dataflow.dflow INFO: Standard output for task 2188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/237ef975-8eee-4b5b-9879-283a1a6f9763_isrForCrosstalkSources_1125947_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/237ef975-8eee-4b5b-9879-283a1a6f9763_isrForCrosstalkSources_1125947_29.stderr parsl.dataflow.dflow DEBUG: Task 2189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2189 has memoization hash 22c8ae1ecef5222ebb37d714815f02ee parsl.dataflow.memoization INFO: Task 2189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2189 try 0 launched on executor multi with executor id 2190 parsl.dataflow.dflow INFO: Standard output for task 2189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c977e1ca-891d-447a-931c-0bde7de6ae42_isrForCrosstalkSources_1125916_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c977e1ca-891d-447a-931c-0bde7de6ae42_isrForCrosstalkSources_1125916_47.stderr parsl.dataflow.dflow DEBUG: Task 2190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2190 has memoization hash 5b0672bac320966c62ab323fa4be2528 parsl.dataflow.memoization INFO: Task 2190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2190 try 0 launched on executor multi with executor id 2191 parsl.dataflow.dflow INFO: Standard output for task 2190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/77252619-df92-4a89-8f6a-d0abe80cdde0_isrForCrosstalkSources_1125925_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/77252619-df92-4a89-8f6a-d0abe80cdde0_isrForCrosstalkSources_1125925_52.stderr parsl.dataflow.dflow DEBUG: Task 2191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2191 has memoization hash b797d33edc78578ab13fb6c4863842e9 parsl.dataflow.memoization INFO: Task 2191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2191 try 0 launched on executor multi with executor id 2192 parsl.dataflow.dflow INFO: Standard output for task 2191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/d6eb359c-6c81-4f96-a914-2719b5af1560_isrForCrosstalkSources_1125921_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/d6eb359c-6c81-4f96-a914-2719b5af1560_isrForCrosstalkSources_1125921_34.stderr parsl.dataflow.dflow DEBUG: Task 2192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2192 has memoization hash 9a5a94bdd9126ea6b8c9e60d4408d3d3 parsl.dataflow.memoization INFO: Task 2192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2192 try 0 launched on executor multi with executor id 2193 parsl.dataflow.dflow INFO: Standard output for task 2192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/9e9e2e35-67a1-4b1d-8133-1456af911938_isrForCrosstalkSources_1125905_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/9e9e2e35-67a1-4b1d-8133-1456af911938_isrForCrosstalkSources_1125905_15.stderr parsl.dataflow.dflow DEBUG: Task 2193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2193 has memoization hash e6898fb201110cb519c06b08562a6ee7 parsl.dataflow.memoization INFO: Task 2193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2193 try 0 launched on executor multi with executor id 2194 parsl.dataflow.dflow INFO: Standard output for task 2193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/7e2c6163-f8a9-42ec-bc87-facfcc9fbac6_isrForCrosstalkSources_1125943_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/7e2c6163-f8a9-42ec-bc87-facfcc9fbac6_isrForCrosstalkSources_1125943_59.stderr parsl.dataflow.dflow DEBUG: Task 2194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2194 has memoization hash 16d490efec073ccccfdf78d578d2879c parsl.dataflow.memoization INFO: Task 2194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2194 try 0 launched on executor multi with executor id 2195 parsl.dataflow.dflow INFO: Standard output for task 2194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/9510ceaf-98e4-4229-aae0-756886a1a0de_isrForCrosstalkSources_1125940_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/9510ceaf-98e4-4229-aae0-756886a1a0de_isrForCrosstalkSources_1125940_28.stderr parsl.dataflow.dflow DEBUG: Task 2195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2195 has memoization hash 99fac64849f1ff6d9313a0f29c5fc06e parsl.dataflow.memoization INFO: Task 2195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2195 try 0 launched on executor multi with executor id 2196 parsl.dataflow.dflow INFO: Standard output for task 2195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/35cbc9b3-f0bd-449a-a847-d3bd03d8ad77_isrForCrosstalkSources_1125940_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/35cbc9b3-f0bd-449a-a847-d3bd03d8ad77_isrForCrosstalkSources_1125940_20.stderr parsl.dataflow.dflow DEBUG: Task 2196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2196 has memoization hash 5b2aeebe46aaa829d84e7bef21c98102 parsl.dataflow.memoization INFO: Task 2196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2196 try 0 launched on executor multi with executor id 2197 parsl.dataflow.dflow INFO: Standard output for task 2196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/7f40f5d0-5cab-428e-9290-5e1d46e17dc6_isrForCrosstalkSources_1125939_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/7f40f5d0-5cab-428e-9290-5e1d46e17dc6_isrForCrosstalkSources_1125939_2.stderr parsl.dataflow.dflow DEBUG: Task 2197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2197 has memoization hash 2e5ff416260ef0adf55676ad5893f00f parsl.dataflow.memoization INFO: Task 2197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2197 try 0 launched on executor multi with executor id 2198 parsl.dataflow.dflow INFO: Standard output for task 2197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/dd823f63-7b89-451d-a99d-6dded3c04938_isrForCrosstalkSources_1125898_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/dd823f63-7b89-451d-a99d-6dded3c04938_isrForCrosstalkSources_1125898_46.stderr parsl.dataflow.dflow DEBUG: Task 2198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2198 has memoization hash 3b21b148a1686caa9331f00d3f2a62b5 parsl.dataflow.memoization INFO: Task 2198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2198 try 0 launched on executor multi with executor id 2199 parsl.dataflow.dflow INFO: Standard output for task 2198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/6f9312c6-4c51-4c78-8fd8-fda45b494e7d_isrForCrosstalkSources_1125891_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/6f9312c6-4c51-4c78-8fd8-fda45b494e7d_isrForCrosstalkSources_1125891_46.stderr parsl.dataflow.dflow DEBUG: Task 2199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2199 has memoization hash b90e375a2f2fec4ef979dc137b052f7e parsl.dataflow.memoization INFO: Task 2199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2199 try 0 launched on executor multi with executor id 2200 parsl.dataflow.dflow INFO: Standard output for task 2199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/c8d975b6-7ab1-46be-9de7-1125ddebbc42_isrForCrosstalkSources_1125947_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/c8d975b6-7ab1-46be-9de7-1125ddebbc42_isrForCrosstalkSources_1125947_3.stderr parsl.dataflow.dflow DEBUG: Task 2200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2200 has memoization hash 39253c3af398e7e5d06d6f4e865b51c5 parsl.dataflow.memoization INFO: Task 2200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2200 try 0 launched on executor multi with executor id 2201 parsl.dataflow.dflow INFO: Standard output for task 2200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/6837769e-7b51-42ec-92af-2088c0bf3af5_isrForCrosstalkSources_1125945_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/6837769e-7b51-42ec-92af-2088c0bf3af5_isrForCrosstalkSources_1125945_19.stderr parsl.dataflow.dflow DEBUG: Task 2201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2201 has memoization hash 85bde8a9d54ee5cebaa04fbb40f01d5b parsl.dataflow.memoization INFO: Task 2201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73a6cbf49f5ef46e847f42c4f79aee8b with result from task 0 parsl.dataflow.dflow INFO: Parsl task 2201 try 0 launched on executor multi with executor id 2202 parsl.dataflow.dflow INFO: Standard output for task 2201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/8f4d6fcb-287a-4026-aa28-74642537911b_isrForCrosstalkSources_1125937_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/8f4d6fcb-287a-4026-aa28-74642537911b_isrForCrosstalkSources_1125937_17.stderr parsl.dataflow.dflow DEBUG: Task 2202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2202 has memoization hash a24b9797b12b9918773e8e4cc3bebb89 parsl.dataflow.memoization INFO: Task 2202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2202 try 0 launched on executor multi with executor id 2203 parsl.dataflow.dflow INFO: Standard output for task 2202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/1799fa02-93d3-4c38-baf9-29cec97490ad_isrForCrosstalkSources_1125905_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/1799fa02-93d3-4c38-baf9-29cec97490ad_isrForCrosstalkSources_1125905_27.stderr parsl.dataflow.dflow DEBUG: Task 2203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2203 has memoization hash af106c970167349646b87ec0563d139d parsl.dataflow.memoization INFO: Task 2203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2203 try 0 launched on executor multi with executor id 2204 parsl.dataflow.dflow INFO: Standard output for task 2203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/8f69aa2f-00db-45b1-aea9-bfa2b2e924d3_isrForCrosstalkSources_1125890_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/8f69aa2f-00db-45b1-aea9-bfa2b2e924d3_isrForCrosstalkSources_1125890_26.stderr parsl.dataflow.dflow DEBUG: Task 2204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2204 has memoization hash 6fbf7d63555bc8834483a9a36a5a4f76 parsl.dataflow.memoization INFO: Task 2204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2204 try 0 launched on executor multi with executor id 2205 parsl.dataflow.dflow INFO: Standard output for task 2204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/4c2f2fa4-b709-4f32-9621-4eb481d278df_isrForCrosstalkSources_1125894_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/4c2f2fa4-b709-4f32-9621-4eb481d278df_isrForCrosstalkSources_1125894_6.stderr parsl.dataflow.dflow DEBUG: Task 2205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2205 has memoization hash 392baefeb2e6c3f45beb816d7448d453 parsl.dataflow.memoization INFO: Task 2205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Parsl task 2205 try 0 launched on executor multi with executor id 2206 parsl.dataflow.dflow INFO: Standard output for task 2205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/445000a4-8d59-4063-b12a-90c41fcf8f59_isrForCrosstalkSources_1125897_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/445000a4-8d59-4063-b12a-90c41fcf8f59_isrForCrosstalkSources_1125897_39.stderr parsl.dataflow.dflow DEBUG: Task 2206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2206 has memoization hash 328ab20a9d65a98056dc9648f4856320 parsl.dataflow.memoization INFO: Task 2206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2206 try 0 launched on executor multi with executor id 2207 parsl.dataflow.dflow INFO: Standard output for task 2206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/cba191d9-2c1f-458d-9116-c17cda5c45f0_isrForCrosstalkSources_1125935_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/cba191d9-2c1f-458d-9116-c17cda5c45f0_isrForCrosstalkSources_1125935_30.stderr parsl.dataflow.dflow DEBUG: Task 2207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2207 has memoization hash c14d25756ad42c367fee791edd7c1c53 parsl.dataflow.memoization INFO: Task 2207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/98bdeb89-2533-40dd-b017-91c8bd9198ee_isrForCrosstalkSources_1125906_45.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/98bdeb89-2533-40dd-b017-91c8bd9198ee_isrForCrosstalkSources_1125906_45.stderr parsl.dataflow.dflow INFO: Parsl task 2207 try 0 launched on executor multi with executor id 2208 parsl.dataflow.dflow INFO: Standard output for task 2207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/fa62127b-a001-4f82-a40f-7c0b4356163d_isrForCrosstalkSources_1125905_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/fa62127b-a001-4f82-a40f-7c0b4356163d_isrForCrosstalkSources_1125905_11.stderr parsl.dataflow.dflow DEBUG: Task 2208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2208 has memoization hash a5016ecaafa12d601912f76a45bd2425 parsl.dataflow.memoization INFO: Task 2208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2208 try 0 launched on executor multi with executor id 2209 parsl.dataflow.dflow INFO: Standard output for task 2208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/6dd00cd1-74f6-4954-84ab-854a84ce07d3_isrForCrosstalkSources_1125947_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/6dd00cd1-74f6-4954-84ab-854a84ce07d3_isrForCrosstalkSources_1125947_9.stderr parsl.dataflow.dflow DEBUG: Task 2209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2209 has memoization hash 9bb4beeb4579d026c254a46fecb72904 parsl.dataflow.memoization INFO: Task 2209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2209 try 0 launched on executor multi with executor id 2210 parsl.dataflow.dflow INFO: Standard output for task 2209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/eb05f2c9-5f4e-489f-88ac-1157c0834d5b_isrForCrosstalkSources_1125936_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/eb05f2c9-5f4e-489f-88ac-1157c0834d5b_isrForCrosstalkSources_1125936_42.stderr parsl.dataflow.dflow DEBUG: Task 2210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2210 has memoization hash e4f2c13452f047471e81a23e44ff7be3 parsl.dataflow.memoization INFO: Task 2210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2210 try 0 launched on executor multi with executor id 2211 parsl.dataflow.dflow INFO: Standard output for task 2210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/0d050859-d80a-42d9-98bf-0ec0aecb9bcd_isrForCrosstalkSources_1125925_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/0d050859-d80a-42d9-98bf-0ec0aecb9bcd_isrForCrosstalkSources_1125925_28.stderr parsl.dataflow.dflow DEBUG: Task 2211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2211 has memoization hash 9acfc366b6fb08a429771580c49c7e86 parsl.dataflow.memoization INFO: Task 2211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2211 try 0 launched on executor multi with executor id 2212 parsl.dataflow.dflow INFO: Standard output for task 2211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/28f0a461-776a-43a5-9292-9f5167fe956e_isrForCrosstalkSources_1125930_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/28f0a461-776a-43a5-9292-9f5167fe956e_isrForCrosstalkSources_1125930_54.stderr parsl.dataflow.dflow DEBUG: Task 2212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2212 has memoization hash 491899dcccd2b1e9c96d8f9671016d4f parsl.dataflow.memoization INFO: Task 2212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2212 try 0 launched on executor multi with executor id 2213 parsl.dataflow.dflow INFO: Standard output for task 2212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/27637ab8-70fe-4286-b636-2e8297ecde66_isrForCrosstalkSources_1125922_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/27637ab8-70fe-4286-b636-2e8297ecde66_isrForCrosstalkSources_1125922_26.stderr parsl.dataflow.dflow DEBUG: Task 2213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2213 has memoization hash 33f260e4165130d0a0af4f696438461c parsl.dataflow.memoization INFO: Task 2213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2213 try 0 launched on executor multi with executor id 2214 parsl.dataflow.dflow INFO: Standard output for task 2213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/3e578b30-e372-4e90-b4f6-fc30b7149e13_isrForCrosstalkSources_1125922_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/3e578b30-e372-4e90-b4f6-fc30b7149e13_isrForCrosstalkSources_1125922_8.stderr parsl.dataflow.dflow DEBUG: Task 2214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2214 has memoization hash 01a23953abea283eeb21e273c9a2876f parsl.dataflow.memoization INFO: Task 2214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2214 try 0 launched on executor multi with executor id 2215 parsl.dataflow.dflow INFO: Standard output for task 2214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/cb377207-5885-4fe0-9f08-6fff2f233371_isrForCrosstalkSources_1125917_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/cb377207-5885-4fe0-9f08-6fff2f233371_isrForCrosstalkSources_1125917_47.stderr parsl.dataflow.dflow DEBUG: Task 2215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2215 has memoization hash cd10780e80900825a815b4e3f9e49a53 parsl.dataflow.memoization INFO: Task 2215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2215 try 0 launched on executor multi with executor id 2216 parsl.dataflow.dflow INFO: Standard output for task 2215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/0baa5af4-0862-41b3-a171-af35fa38e50f_isrForCrosstalkSources_1125939_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/0baa5af4-0862-41b3-a171-af35fa38e50f_isrForCrosstalkSources_1125939_9.stderr parsl.dataflow.dflow DEBUG: Task 2216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2216 has memoization hash 8eabcc6eaa95d32bd7b40a2187be61a2 parsl.dataflow.memoization INFO: Task 2216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2216 try 0 launched on executor multi with executor id 2217 parsl.dataflow.dflow INFO: Standard output for task 2216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/c3aae125-0913-4a35-9f46-09377b2320d1_isrForCrosstalkSources_1125896_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/c3aae125-0913-4a35-9f46-09377b2320d1_isrForCrosstalkSources_1125896_7.stderr parsl.dataflow.dflow DEBUG: Task 2217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2217 has memoization hash 23309571569f5e3c2d61727a5c34ca67 parsl.dataflow.memoization INFO: Task 2217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2217 try 0 launched on executor multi with executor id 2218 parsl.dataflow.dflow INFO: Standard output for task 2217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/8e55cff0-9911-4d5b-aa61-31f77986a7ae_isrForCrosstalkSources_1125916_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/8e55cff0-9911-4d5b-aa61-31f77986a7ae_isrForCrosstalkSources_1125916_17.stderr parsl.dataflow.dflow DEBUG: Task 2218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2218 has memoization hash ee99af57fd909aa631df88f2bd2cfc14 parsl.dataflow.memoization INFO: Task 2218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2218 try 0 launched on executor multi with executor id 2219 parsl.dataflow.dflow INFO: Standard output for task 2218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/f7560a04-07c3-4878-8da5-6a03a0856eaf_isrForCrosstalkSources_1125898_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/f7560a04-07c3-4878-8da5-6a03a0856eaf_isrForCrosstalkSources_1125898_45.stderr parsl.dataflow.dflow DEBUG: Task 2219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2219 has memoization hash b640a27a92a06de8bdf2aec28f521ffd parsl.dataflow.memoization INFO: Task 2219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2219 try 0 launched on executor multi with executor id 2220 parsl.dataflow.dflow INFO: Standard output for task 2219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/c8b29878-a519-4bc2-8f4d-646eb4341469_isrForCrosstalkSources_1125919_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/c8b29878-a519-4bc2-8f4d-646eb4341469_isrForCrosstalkSources_1125919_54.stderr parsl.dataflow.dflow DEBUG: Task 2220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2220 has memoization hash c872779d27ac8f822ef7e5952579b5e0 parsl.dataflow.memoization INFO: Task 2220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2220 try 0 launched on executor multi with executor id 2221 parsl.dataflow.dflow INFO: Standard output for task 2220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/2782a268-894f-453d-93d5-f7148dfc88bd_isrForCrosstalkSources_1125890_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/2782a268-894f-453d-93d5-f7148dfc88bd_isrForCrosstalkSources_1125890_56.stderr parsl.dataflow.dflow DEBUG: Task 2221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2221 has memoization hash 84dc8540c5fc6401e34e54a684cfd72a parsl.dataflow.memoization INFO: Task 2221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2221 try 0 launched on executor multi with executor id 2222 parsl.dataflow.dflow INFO: Standard output for task 2221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/c093de66-9ed2-41d3-b396-78986b6ce3eb_isrForCrosstalkSources_1125920_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/c093de66-9ed2-41d3-b396-78986b6ce3eb_isrForCrosstalkSources_1125920_54.stderr parsl.dataflow.dflow DEBUG: Task 2222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2222 has memoization hash 3bdaa777cd3037ac330b7ad02a83ed51 parsl.dataflow.memoization INFO: Task 2222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2222 try 0 launched on executor multi with executor id 2223 parsl.dataflow.dflow INFO: Standard output for task 2222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/a232ca2b-5551-45fa-81f4-6f3dc2f782be_isrForCrosstalkSources_1125947_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/a232ca2b-5551-45fa-81f4-6f3dc2f782be_isrForCrosstalkSources_1125947_51.stderr parsl.dataflow.dflow DEBUG: Task 2223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2223 has memoization hash 9b40d10c801c13ab8fdc97eb214118e0 parsl.dataflow.memoization INFO: Task 2223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2223 try 0 launched on executor multi with executor id 2224 parsl.dataflow.dflow INFO: Standard output for task 2223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/291f7102-0635-4fc3-8bc6-a95a95555494_isrForCrosstalkSources_1125923_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/291f7102-0635-4fc3-8bc6-a95a95555494_isrForCrosstalkSources_1125923_62.stderr parsl.dataflow.dflow DEBUG: Task 2224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2224 has memoization hash 77f390d3a82f83097a36625ef6670db8 parsl.dataflow.memoization INFO: Task 2224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2224 try 0 launched on executor multi with executor id 2225 parsl.dataflow.dflow INFO: Standard output for task 2224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/fcc4b907-e812-48b4-b96f-c81727382cd6_isrForCrosstalkSources_1125931_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/fcc4b907-e812-48b4-b96f-c81727382cd6_isrForCrosstalkSources_1125931_60.stderr parsl.dataflow.dflow DEBUG: Task 2225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2225 has memoization hash fd9858e428714659f632fb89dde183ba parsl.dataflow.memoization INFO: Task 2225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2225 try 0 launched on executor multi with executor id 2226 parsl.dataflow.dflow INFO: Standard output for task 2225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/705fc3cd-3374-4623-bc33-0e4ea0d84184_isrForCrosstalkSources_1125939_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/705fc3cd-3374-4623-bc33-0e4ea0d84184_isrForCrosstalkSources_1125939_12.stderr parsl.dataflow.dflow DEBUG: Task 2226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2226 has memoization hash c9d8429f08298ac23815360d8d8c0eff parsl.dataflow.memoization INFO: Task 2226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2226 try 0 launched on executor multi with executor id 2227 parsl.dataflow.dflow INFO: Standard output for task 2226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/7e7e039f-b978-4d0f-8fc9-a8626bed8a24_isrForCrosstalkSources_1125890_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/7e7e039f-b978-4d0f-8fc9-a8626bed8a24_isrForCrosstalkSources_1125890_14.stderr parsl.dataflow.dflow DEBUG: Task 2227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2227 has memoization hash 30175459c87bd790d40dfdbc6e868aad parsl.dataflow.memoization INFO: Task 2227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2227 try 0 launched on executor multi with executor id 2228 parsl.dataflow.dflow INFO: Standard output for task 2227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/b2517d6c-9f4d-4131-9b0b-47825a76290b_isrForCrosstalkSources_1125942_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/b2517d6c-9f4d-4131-9b0b-47825a76290b_isrForCrosstalkSources_1125942_37.stderr parsl.dataflow.dflow DEBUG: Task 2228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2228 has memoization hash 6b525fe7488b81f9f287fd03995cfd32 parsl.dataflow.memoization INFO: Task 2228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2228 try 0 launched on executor multi with executor id 2229 parsl.dataflow.dflow INFO: Standard output for task 2228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/fead0822-7f5c-447e-8ddb-02aea6300e91_isrForCrosstalkSources_1125906_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/fead0822-7f5c-447e-8ddb-02aea6300e91_isrForCrosstalkSources_1125906_13.stderr parsl.dataflow.dflow DEBUG: Task 2229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2229 has memoization hash 7b5aa98adb4d2d410c360f320e583f4f parsl.dataflow.memoization INFO: Task 2229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2229 try 0 launched on executor multi with executor id 2230 parsl.dataflow.dflow INFO: Standard output for task 2229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/dbbded6c-6b7f-4744-82dd-22098b8cf86b_isrForCrosstalkSources_1125898_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/dbbded6c-6b7f-4744-82dd-22098b8cf86b_isrForCrosstalkSources_1125898_7.stderr parsl.dataflow.dflow DEBUG: Task 2230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2230 has memoization hash 81c40cf1573443d42a531bbb073fd4e2 parsl.dataflow.memoization INFO: Task 2230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2230 try 0 launched on executor multi with executor id 2231 parsl.dataflow.dflow INFO: Standard output for task 2230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/71db5e86-189b-4ef2-a181-a8b3ccd89b16_isrForCrosstalkSources_1125945_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/71db5e86-189b-4ef2-a181-a8b3ccd89b16_isrForCrosstalkSources_1125945_57.stderr parsl.dataflow.dflow DEBUG: Task 2231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2231 has memoization hash ea569f91cda12fff8999ff05a2ac37d6 parsl.dataflow.memoization INFO: Task 2231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2231 try 0 launched on executor multi with executor id 2232 parsl.dataflow.dflow INFO: Standard output for task 2231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/0dd5f3d7-e720-4446-a24d-f2eab4b65090_isrForCrosstalkSources_1125931_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/0dd5f3d7-e720-4446-a24d-f2eab4b65090_isrForCrosstalkSources_1125931_3.stderr parsl.dataflow.dflow DEBUG: Task 2232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2232 has memoization hash 2e596e3581d199518151f47930af4144 parsl.dataflow.memoization INFO: Task 2232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2232 try 0 launched on executor multi with executor id 2233 parsl.dataflow.dflow INFO: Standard output for task 2232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/f97b48ac-29d0-4090-9717-d2056eb90e06_isrForCrosstalkSources_1125905_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/f97b48ac-29d0-4090-9717-d2056eb90e06_isrForCrosstalkSources_1125905_30.stderr parsl.dataflow.dflow DEBUG: Task 2233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2233 has memoization hash c3de405aaf6a097059771b1152186aa6 parsl.dataflow.memoization INFO: Task 2233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2233 try 0 launched on executor multi with executor id 2234 parsl.dataflow.dflow INFO: Standard output for task 2233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/1859a85f-753f-40fe-be5b-d0d6ee7caaca_isrForCrosstalkSources_1125908_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/1859a85f-753f-40fe-be5b-d0d6ee7caaca_isrForCrosstalkSources_1125908_50.stderr parsl.dataflow.dflow DEBUG: Task 2234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2234 has memoization hash 4c13a554b7f03edbe71bbcf7f37ca3d5 parsl.dataflow.memoization INFO: Task 2234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2234 try 0 launched on executor multi with executor id 2235 parsl.dataflow.dflow INFO: Standard output for task 2234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/b2b0c46d-2837-4a8b-a0e7-d14c70f9fe45_isrForCrosstalkSources_1125945_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/b2b0c46d-2837-4a8b-a0e7-d14c70f9fe45_isrForCrosstalkSources_1125945_56.stderr parsl.dataflow.dflow DEBUG: Task 2235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2235 has memoization hash b993033872b45556f6aa6fcd8de2f6c9 parsl.dataflow.memoization INFO: Task 2235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2235 try 0 launched on executor multi with executor id 2236 parsl.dataflow.dflow INFO: Standard output for task 2235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/68adccd4-ff84-4dbd-a76c-dc81351204c9_isrForCrosstalkSources_1125923_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/68adccd4-ff84-4dbd-a76c-dc81351204c9_isrForCrosstalkSources_1125923_10.stderr parsl.dataflow.dflow DEBUG: Task 2236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2236 has memoization hash c3554ccea9277cbb61f810da8cbd78f5 parsl.dataflow.memoization INFO: Task 2236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2236 try 0 launched on executor multi with executor id 2237 parsl.dataflow.dflow INFO: Standard output for task 2236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/af867f5c-57b1-43d7-abd1-17247b0933ec_isrForCrosstalkSources_1125915_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/af867f5c-57b1-43d7-abd1-17247b0933ec_isrForCrosstalkSources_1125915_19.stderr parsl.dataflow.dflow DEBUG: Task 2237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2237 has memoization hash b8bf4150a64b461a1d86d0c476c74b44 parsl.dataflow.memoization INFO: Task 2237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2237 try 0 launched on executor multi with executor id 2238 parsl.dataflow.dflow INFO: Standard output for task 2237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/4e2ed624-66b8-444c-adaa-e20ed8c35061_isrForCrosstalkSources_1125909_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/4e2ed624-66b8-444c-adaa-e20ed8c35061_isrForCrosstalkSources_1125909_15.stderr parsl.dataflow.dflow DEBUG: Task 2238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2238 has memoization hash 119986c7c1491a421e48b9899114acab parsl.dataflow.memoization INFO: Task 2238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2238 try 0 launched on executor multi with executor id 2239 parsl.dataflow.dflow INFO: Standard output for task 2238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/0a3c92c7-2b43-4cae-bc08-0fac9cfc4d62_isrForCrosstalkSources_1125932_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/0a3c92c7-2b43-4cae-bc08-0fac9cfc4d62_isrForCrosstalkSources_1125932_40.stderr parsl.dataflow.dflow DEBUG: Task 2239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2239 has memoization hash 51e9eea27e39d9377ca477b42326bb87 parsl.dataflow.memoization INFO: Task 2239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2239 try 0 launched on executor multi with executor id 2240 parsl.dataflow.dflow INFO: Standard output for task 2239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/f18a1aa7-ce89-43a4-bc40-03660a92d57e_isrForCrosstalkSources_1125890_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/f18a1aa7-ce89-43a4-bc40-03660a92d57e_isrForCrosstalkSources_1125890_12.stderr parsl.dataflow.dflow DEBUG: Task 2240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2240 has memoization hash b898f0a4e968506f4254ec7e6764fa1d parsl.dataflow.memoization INFO: Task 2240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2240 try 0 launched on executor multi with executor id 2241 parsl.dataflow.dflow INFO: Standard output for task 2240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/4ec1f2e6-ca22-4615-a82c-f2d52343e221_isrForCrosstalkSources_1125926_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/4ec1f2e6-ca22-4615-a82c-f2d52343e221_isrForCrosstalkSources_1125926_14.stderr parsl.dataflow.dflow DEBUG: Task 2241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2241 has memoization hash c2af11108fc43849f0eaadd7962807f2 parsl.dataflow.memoization INFO: Task 2241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2241 try 0 launched on executor multi with executor id 2242 parsl.dataflow.dflow INFO: Standard output for task 2241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/ffa94ab9-141a-4a4e-b064-809262f5c02d_isrForCrosstalkSources_1125888_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/ffa94ab9-141a-4a4e-b064-809262f5c02d_isrForCrosstalkSources_1125888_12.stderr parsl.dataflow.dflow DEBUG: Task 2242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2242 has memoization hash 919a80cc1eb14b630629ca5f171977ea parsl.dataflow.memoization INFO: Task 2242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2242 try 0 launched on executor multi with executor id 2243 parsl.dataflow.dflow INFO: Standard output for task 2242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/3609bc73-cc3c-4fb6-947e-3f2e7c15f520_isrForCrosstalkSources_1125925_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/3609bc73-cc3c-4fb6-947e-3f2e7c15f520_isrForCrosstalkSources_1125925_58.stderr parsl.dataflow.dflow DEBUG: Task 2243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2243 has memoization hash 55199da927d7846386751ce3d49e72d0 parsl.dataflow.memoization INFO: Task 2243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2243 try 0 launched on executor multi with executor id 2244 parsl.dataflow.dflow INFO: Standard output for task 2243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/22b4f0f4-8351-41b0-9f89-8288d8c4526e_isrForCrosstalkSources_1125922_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/22b4f0f4-8351-41b0-9f89-8288d8c4526e_isrForCrosstalkSources_1125922_39.stderr parsl.dataflow.dflow DEBUG: Task 2244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2244 has memoization hash c65fd24c1b654bb0934dd165a0bfa1a0 parsl.dataflow.memoization INFO: Task 2244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2244 try 0 launched on executor multi with executor id 2245 parsl.dataflow.dflow INFO: Standard output for task 2244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/02c42c5e-cde6-48ba-b8bb-2c7ac7a44d15_isrForCrosstalkSources_1125902_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/02c42c5e-cde6-48ba-b8bb-2c7ac7a44d15_isrForCrosstalkSources_1125902_40.stderr parsl.dataflow.dflow DEBUG: Task 2245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2245 has memoization hash 29707f5bba3009859a1549667970db04 parsl.dataflow.memoization INFO: Task 2245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2245 try 0 launched on executor multi with executor id 2246 parsl.dataflow.dflow INFO: Standard output for task 2245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/46da1284-ce8e-4605-bf87-ebdcd22fc2ac_isrForCrosstalkSources_1125923_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/46da1284-ce8e-4605-bf87-ebdcd22fc2ac_isrForCrosstalkSources_1125923_15.stderr parsl.dataflow.dflow DEBUG: Task 2246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2246 has memoization hash b0130f063338580f5ef97c3697466b4d parsl.dataflow.memoization INFO: Task 2246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2246 try 0 launched on executor multi with executor id 2247 parsl.dataflow.dflow INFO: Standard output for task 2246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/e31a579e-b9ef-4da3-986b-a04cd2a16599_isrForCrosstalkSources_1125923_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/e31a579e-b9ef-4da3-986b-a04cd2a16599_isrForCrosstalkSources_1125923_30.stderr parsl.dataflow.dflow DEBUG: Task 2247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2247 has memoization hash 276dcf7a3758ccffb999c644b81e79cf parsl.dataflow.memoization INFO: Task 2247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2247 try 0 launched on executor multi with executor id 2248 parsl.dataflow.dflow INFO: Standard output for task 2247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/e929ff0f-49b3-4481-bd32-78813f590ea8_isrForCrosstalkSources_1125894_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/e929ff0f-49b3-4481-bd32-78813f590ea8_isrForCrosstalkSources_1125894_25.stderr parsl.dataflow.dflow DEBUG: Task 2248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2248 has memoization hash 887546980c32155d49215fd16fab5a29 parsl.dataflow.memoization INFO: Task 2248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a398b3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2248 try 0 launched on executor multi with executor id 2249 parsl.dataflow.dflow INFO: Standard output for task 2248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/59e2a740-e6bf-41f6-8280-1a663bf1a327_isrForCrosstalkSources_1125934_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/59e2a740-e6bf-41f6-8280-1a663bf1a327_isrForCrosstalkSources_1125934_38.stderr parsl.dataflow.dflow DEBUG: Task 2249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2249 has memoization hash df850f894a5de4d2ce01410db0765195 parsl.dataflow.memoization INFO: Task 2249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397480e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2249 try 0 launched on executor multi with executor id 2250 parsl.dataflow.dflow INFO: Standard output for task 2249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/bd1b1e3d-41a0-4574-8237-fcadb3bd0721_isrForCrosstalkSources_1125927_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/bd1b1e3d-41a0-4574-8237-fcadb3bd0721_isrForCrosstalkSources_1125927_42.stderr parsl.dataflow.dflow DEBUG: Task 2250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2250 has memoization hash 5dbfd04bb1665229bc12183fa11ce878 parsl.dataflow.memoization INFO: Task 2250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2250 try 0 launched on executor multi with executor id 2251 parsl.dataflow.dflow INFO: Standard output for task 2250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/698fd1f7-7bfc-4154-af3d-6eea431ae838_isrForCrosstalkSources_1125910_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/698fd1f7-7bfc-4154-af3d-6eea431ae838_isrForCrosstalkSources_1125910_37.stderr parsl.dataflow.dflow DEBUG: Task 2251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2251 has memoization hash d1011b59d34bb5588ed4948f43e13832 parsl.dataflow.memoization INFO: Task 2251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2251 try 0 launched on executor multi with executor id 2252 parsl.dataflow.dflow INFO: Standard output for task 2251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/3b64d45b-4c56-4c25-a20c-0220ba47f1bc_isrForCrosstalkSources_1125935_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/3b64d45b-4c56-4c25-a20c-0220ba47f1bc_isrForCrosstalkSources_1125935_22.stderr parsl.dataflow.dflow DEBUG: Task 2252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2252 has memoization hash 1cc504d0cb887396acafb09841e59fb9 parsl.dataflow.memoization INFO: Task 2252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2252 try 0 launched on executor multi with executor id 2253 parsl.dataflow.dflow INFO: Standard output for task 2252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0855394c-92ca-4142-a7c6-586093b468e3_isrForCrosstalkSources_1125914_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0855394c-92ca-4142-a7c6-586093b468e3_isrForCrosstalkSources_1125914_41.stderr parsl.dataflow.dflow DEBUG: Task 2253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2253 has memoization hash a2cef90c258132decd43f22d3d330979 parsl.dataflow.memoization INFO: Task 2253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397482c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2253 try 0 launched on executor multi with executor id 2254 parsl.dataflow.dflow INFO: Standard output for task 2253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/73523e99-e0e7-4470-a814-aec7cefb0f5b_isrForCrosstalkSources_1125925_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/73523e99-e0e7-4470-a814-aec7cefb0f5b_isrForCrosstalkSources_1125925_6.stderr parsl.dataflow.dflow DEBUG: Task 2254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2254 has memoization hash 31e1abeb8be6040fd77222a3a4201dc6 parsl.dataflow.memoization INFO: Task 2254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2254 try 0 launched on executor multi with executor id 2255 parsl.dataflow.dflow INFO: Standard output for task 2254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/632fab6f-4401-4cf6-9acf-352501e22b32_isrForCrosstalkSources_1125904_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/632fab6f-4401-4cf6-9acf-352501e22b32_isrForCrosstalkSources_1125904_31.stderr parsl.dataflow.dflow DEBUG: Task 2255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2255 has memoization hash 4fbc671f1ae567449d32780b0cc97a8a parsl.dataflow.memoization INFO: Task 2255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397484a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2255 try 0 launched on executor multi with executor id 2256 parsl.dataflow.dflow INFO: Standard output for task 2255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/21697b4e-df33-4fc5-90df-25f8f41290b4_isrForCrosstalkSources_1125890_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/21697b4e-df33-4fc5-90df-25f8f41290b4_isrForCrosstalkSources_1125890_54.stderr parsl.dataflow.dflow DEBUG: Task 2256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2256 has memoization hash 843bf13ecb4ae6dae420ed654f653e3b parsl.dataflow.memoization INFO: Task 2256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2256 try 0 launched on executor multi with executor id 2257 parsl.dataflow.dflow INFO: Standard output for task 2256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/e8369035-ccc8-496e-b5f7-744bdf6d6752_isrForCrosstalkSources_1125941_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/e8369035-ccc8-496e-b5f7-744bdf6d6752_isrForCrosstalkSources_1125941_50.stderr parsl.dataflow.dflow DEBUG: Task 2257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2257 has memoization hash 0f5e348d668616d439c965416f73c1e3 parsl.dataflow.memoization INFO: Task 2257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2257 try 0 launched on executor multi with executor id 2258 parsl.dataflow.dflow INFO: Standard output for task 2257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/4bff4953-7c48-4312-9892-d295c27b9a5a_isrForCrosstalkSources_1125895_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/4bff4953-7c48-4312-9892-d295c27b9a5a_isrForCrosstalkSources_1125895_21.stderr parsl.dataflow.dflow DEBUG: Task 2258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2258 has memoization hash 295e01e91d547f5f4a75d0975c47fbfa parsl.dataflow.memoization INFO: Task 2258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397485e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2258 try 0 launched on executor multi with executor id 2259 parsl.dataflow.dflow INFO: Standard output for task 2258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/c8bacd72-6de5-4afa-93f6-1838d804fb00_isrForCrosstalkSources_1125929_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/c8bacd72-6de5-4afa-93f6-1838d804fb00_isrForCrosstalkSources_1125929_8.stderr parsl.dataflow.dflow DEBUG: Task 2259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2259 has memoization hash 6b92fb6741ee855a4032be7cfa44a721 parsl.dataflow.memoization INFO: Task 2259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2259 try 0 launched on executor multi with executor id 2260 parsl.dataflow.dflow INFO: Standard output for task 2259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/32ffbda3-cd92-4317-912b-1522ab435a3f_isrForCrosstalkSources_1125888_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/32ffbda3-cd92-4317-912b-1522ab435a3f_isrForCrosstalkSources_1125888_15.stderr parsl.dataflow.dflow DEBUG: Task 2260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2260 has memoization hash cea27635e9c325d7603afb6d7daaf485 parsl.dataflow.memoization INFO: Task 2260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397487c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2260 try 0 launched on executor multi with executor id 2261 parsl.dataflow.dflow INFO: Standard output for task 2260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/0c564588-c751-4677-8ef0-dcedbeee7f4a_isrForCrosstalkSources_1125922_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/0c564588-c751-4677-8ef0-dcedbeee7f4a_isrForCrosstalkSources_1125922_34.stderr parsl.dataflow.dflow DEBUG: Task 2261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2261 has memoization hash a3a4eb290e659e9b70c75dd22164536b parsl.dataflow.memoization INFO: Task 2261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2261 try 0 launched on executor multi with executor id 2262 parsl.dataflow.dflow INFO: Standard output for task 2261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/106f4519-82ff-4e32-ab7d-72ed5149e171_isrForCrosstalkSources_1125940_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/106f4519-82ff-4e32-ab7d-72ed5149e171_isrForCrosstalkSources_1125940_35.stderr parsl.dataflow.dflow DEBUG: Task 2262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2262 has memoization hash b9c96577675a3eae008123d49c0b07ad parsl.dataflow.memoization INFO: Task 2262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397489a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2262 try 0 launched on executor multi with executor id 2263 parsl.dataflow.dflow INFO: Standard output for task 2262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/b50eeb99-37f1-4718-b4d9-6229db89fa19_isrForCrosstalkSources_1125895_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/b50eeb99-37f1-4718-b4d9-6229db89fa19_isrForCrosstalkSources_1125895_57.stderr parsl.dataflow.dflow DEBUG: Task 2263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2263 has memoization hash b2377640cf5dc91ecdd86fd5ea9fb796 parsl.dataflow.memoization INFO: Task 2263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2263 try 0 launched on executor multi with executor id 2264 parsl.dataflow.dflow INFO: Standard output for task 2263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/a93c81cb-185b-43f9-9c92-6a52f0139315_isrForCrosstalkSources_1125889_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/a93c81cb-185b-43f9-9c92-6a52f0139315_isrForCrosstalkSources_1125889_21.stderr parsl.dataflow.dflow DEBUG: Task 2264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2264 has memoization hash 2e2d3c6b2edc5c019ed06af33eebf3c2 parsl.dataflow.memoization INFO: Task 2264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2264 try 0 launched on executor multi with executor id 2265 parsl.dataflow.dflow INFO: Standard output for task 2264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/05ac66f5-7ac2-49f5-8cfb-a975ecd3e360_isrForCrosstalkSources_1125941_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/05ac66f5-7ac2-49f5-8cfb-a975ecd3e360_isrForCrosstalkSources_1125941_35.stderr parsl.dataflow.dflow DEBUG: Task 2265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2265 has memoization hash a3770f217811ad91533e95b52f30b461 parsl.dataflow.memoization INFO: Task 2265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2265 try 0 launched on executor multi with executor id 2266 parsl.dataflow.dflow INFO: Standard output for task 2265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e5901df3-5814-4214-9d94-c14e00fdc230_isrForCrosstalkSources_1125929_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e5901df3-5814-4214-9d94-c14e00fdc230_isrForCrosstalkSources_1125929_55.stderr parsl.dataflow.dflow DEBUG: Task 2266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2266 has memoization hash 7b1b6bc59f1839a0880caa80685a1f94 parsl.dataflow.memoization INFO: Task 2266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2266 try 0 launched on executor multi with executor id 2267 parsl.dataflow.dflow INFO: Standard output for task 2266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/264aded8-f06c-45ca-bd90-7a64da8a04b4_isrForCrosstalkSources_1125938_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/264aded8-f06c-45ca-bd90-7a64da8a04b4_isrForCrosstalkSources_1125938_24.stderr parsl.dataflow.dflow DEBUG: Task 2267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2267 has memoization hash 96ba0ee97477ce44d561b67ef1d672b1 parsl.dataflow.memoization INFO: Task 2267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2267 try 0 launched on executor multi with executor id 2268 parsl.dataflow.dflow INFO: Standard output for task 2267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/696a8951-2349-4f6f-a318-86a78ac2f77e_isrForCrosstalkSources_1125904_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/696a8951-2349-4f6f-a318-86a78ac2f77e_isrForCrosstalkSources_1125904_37.stderr parsl.dataflow.dflow DEBUG: Task 2268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2268 has memoization hash 9dbfc1a1aa6c15927254b20d85ef2727 parsl.dataflow.memoization INFO: Task 2268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2268 try 0 launched on executor multi with executor id 2269 parsl.dataflow.dflow INFO: Standard output for task 2268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/dbf09398-dbb1-4903-9645-40abba962ac2_isrForCrosstalkSources_1125900_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/dbf09398-dbb1-4903-9645-40abba962ac2_isrForCrosstalkSources_1125900_7.stderr parsl.dataflow.dflow DEBUG: Task 2269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2269 has memoization hash 008711a3379cf30f423ce2dc5d22f2d2 parsl.dataflow.memoization INFO: Task 2269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2269 try 0 launched on executor multi with executor id 2270 parsl.dataflow.dflow INFO: Standard output for task 2269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/726a7f5a-e500-4536-b33e-0f296ef12943_isrForCrosstalkSources_1125922_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/726a7f5a-e500-4536-b33e-0f296ef12943_isrForCrosstalkSources_1125922_6.stderr parsl.dataflow.dflow DEBUG: Task 2270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2270 has memoization hash f0b0e4b68c4c61501330e6afd1ec558b parsl.dataflow.memoization INFO: Task 2270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2270 try 0 launched on executor multi with executor id 2271 parsl.dataflow.dflow INFO: Standard output for task 2270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/84849fdb-565c-4cec-9dbc-597b7c7c8b8f_isrForCrosstalkSources_1125925_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/84849fdb-565c-4cec-9dbc-597b7c7c8b8f_isrForCrosstalkSources_1125925_25.stderr parsl.dataflow.dflow DEBUG: Task 2271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2271 has memoization hash b53658c83fb15097f0e388ef159f2e52 parsl.dataflow.memoization INFO: Task 2271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2271 try 0 launched on executor multi with executor id 2272 parsl.dataflow.dflow INFO: Standard output for task 2271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/49028992-d6f3-47a9-bfdb-1f912f0435b9_isrForCrosstalkSources_1125917_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/49028992-d6f3-47a9-bfdb-1f912f0435b9_isrForCrosstalkSources_1125917_21.stderr parsl.dataflow.dflow DEBUG: Task 2272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2272 has memoization hash 7b854a8f1cc7ca1b9eaf22914025b010 parsl.dataflow.memoization INFO: Task 2272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2272 try 0 launched on executor multi with executor id 2273 parsl.dataflow.dflow INFO: Standard output for task 2272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/22b9e9cd-36cb-4d5e-8ae8-7717dd68481f_isrForCrosstalkSources_1125900_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/22b9e9cd-36cb-4d5e-8ae8-7717dd68481f_isrForCrosstalkSources_1125900_12.stderr parsl.dataflow.dflow DEBUG: Task 2273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2273 has memoization hash fe478c6ea0343d7164154a9e4f268d46 parsl.dataflow.memoization INFO: Task 2273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2273 try 0 launched on executor multi with executor id 2274 parsl.dataflow.dflow INFO: Standard output for task 2273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/126e08cd-97cc-4589-9e30-cd9cb9f806d2_isrForCrosstalkSources_1125893_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/126e08cd-97cc-4589-9e30-cd9cb9f806d2_isrForCrosstalkSources_1125893_39.stderr parsl.dataflow.dflow DEBUG: Task 2274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2274 has memoization hash 14e752c9db165e7cd142ab9e70c47b52 parsl.dataflow.memoization INFO: Task 2274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2274 try 0 launched on executor multi with executor id 2275 parsl.dataflow.dflow INFO: Standard output for task 2274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/03ddd70e-0cad-4f6a-933d-702eba20b03a_isrForCrosstalkSources_1125933_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/03ddd70e-0cad-4f6a-933d-702eba20b03a_isrForCrosstalkSources_1125933_51.stderr parsl.dataflow.dflow DEBUG: Task 2275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2275 has memoization hash ff4fc90737b95a2749de8cf587696e4e parsl.dataflow.memoization INFO: Task 2275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2275 try 0 launched on executor multi with executor id 2276 parsl.dataflow.dflow INFO: Standard output for task 2275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/ea503690-4d94-4148-8ea5-5274a53e28b0_isrForCrosstalkSources_1125940_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/ea503690-4d94-4148-8ea5-5274a53e28b0_isrForCrosstalkSources_1125940_42.stderr parsl.dataflow.dflow DEBUG: Task 2276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2276 has memoization hash 8c9dd799bb372007c654e8b6e0087c9c parsl.dataflow.memoization INFO: Task 2276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397491c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2276 try 0 launched on executor multi with executor id 2277 parsl.dataflow.dflow INFO: Standard output for task 2276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/dcdf2ef5-ca11-42f9-9d80-f5fbf438bb2d_isrForCrosstalkSources_1125922_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/dcdf2ef5-ca11-42f9-9d80-f5fbf438bb2d_isrForCrosstalkSources_1125922_49.stderr parsl.dataflow.dflow DEBUG: Task 2277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2277 has memoization hash 81e895717d6a09165dc04dff2d0b52a1 parsl.dataflow.memoization INFO: Task 2277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2277 try 0 launched on executor multi with executor id 2278 parsl.dataflow.dflow INFO: Standard output for task 2277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/481c27ed-d7d3-4ac2-ac41-d0547ef61d0b_isrForCrosstalkSources_1125893_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/481c27ed-d7d3-4ac2-ac41-d0547ef61d0b_isrForCrosstalkSources_1125893_20.stderr parsl.dataflow.dflow DEBUG: Task 2278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2278 has memoization hash 207dfc4f957e73b3ac4a6e77d974417d parsl.dataflow.memoization INFO: Task 2278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2278 try 0 launched on executor multi with executor id 2279 parsl.dataflow.dflow INFO: Standard output for task 2278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/133f2afd-c1dc-45db-858f-74696f0c8453_isrForCrosstalkSources_1125941_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/133f2afd-c1dc-45db-858f-74696f0c8453_isrForCrosstalkSources_1125941_30.stderr parsl.dataflow.dflow DEBUG: Task 2279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2279 has memoization hash b1a76341370a77e849f19621221c5937 parsl.dataflow.memoization INFO: Task 2279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397493a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2279 try 0 launched on executor multi with executor id 2280 parsl.dataflow.dflow INFO: Standard output for task 2279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/5f8cf3a8-4b5b-47f9-9f67-c806e0b5b34f_isrForCrosstalkSources_1125893_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/5f8cf3a8-4b5b-47f9-9f67-c806e0b5b34f_isrForCrosstalkSources_1125893_51.stderr parsl.dataflow.dflow DEBUG: Task 2280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2280 has memoization hash 05f3b8f8330dcb0a1dfcbeb88fd22b5d parsl.dataflow.memoization INFO: Task 2280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2280 try 0 launched on executor multi with executor id 2281 parsl.dataflow.dflow INFO: Standard output for task 2280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b2035a2a-10b5-4e8a-b363-2b1d18443eb4_isrForCrosstalkSources_1125938_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b2035a2a-10b5-4e8a-b363-2b1d18443eb4_isrForCrosstalkSources_1125938_5.stderr parsl.dataflow.dflow DEBUG: Task 2281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2281 has memoization hash c693f483a7b34486fe20e672627b36b7 parsl.dataflow.memoization INFO: Task 2281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397494e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2281 try 0 launched on executor multi with executor id 2282 parsl.dataflow.dflow INFO: Standard output for task 2281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/3581eb79-7415-4797-9b40-0deb2466b66d_isrForCrosstalkSources_1125918_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/3581eb79-7415-4797-9b40-0deb2466b66d_isrForCrosstalkSources_1125918_11.stderr parsl.dataflow.dflow DEBUG: Task 2282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2282 has memoization hash c95325c851d2a22232b40f0f35450116 parsl.dataflow.memoization INFO: Task 2282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2282 try 0 launched on executor multi with executor id 2283 parsl.dataflow.dflow INFO: Standard output for task 2282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/501cd0f7-d786-45c1-83c5-9c518e53e517_isrForCrosstalkSources_1125903_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/501cd0f7-d786-45c1-83c5-9c518e53e517_isrForCrosstalkSources_1125903_53.stderr parsl.dataflow.dflow DEBUG: Task 2283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2283 has memoization hash 09203cc98dd03e6afb8b74eb7d15d2ce parsl.dataflow.memoization INFO: Task 2283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2283 try 0 launched on executor multi with executor id 2284 parsl.dataflow.dflow INFO: Standard output for task 2283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/6c9b3c9f-3858-496f-9df4-9199cc7a87a1_isrForCrosstalkSources_1125942_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/6c9b3c9f-3858-496f-9df4-9199cc7a87a1_isrForCrosstalkSources_1125942_18.stderr parsl.dataflow.dflow DEBUG: Task 2284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2284 has memoization hash 1abf2b5b15c55218d5fb0c3a836a21b9 parsl.dataflow.memoization INFO: Task 2284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397496c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2284 try 0 launched on executor multi with executor id 2285 parsl.dataflow.dflow INFO: Standard output for task 2284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/2b69771d-02ce-45cf-b00f-4dc8be7c44b1_isrForCrosstalkSources_1125936_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/2b69771d-02ce-45cf-b00f-4dc8be7c44b1_isrForCrosstalkSources_1125936_52.stderr parsl.dataflow.dflow DEBUG: Task 2285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2285 has memoization hash 88d545926b97bda0ec4806fdd7e9b256 parsl.dataflow.memoization INFO: Task 2285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2285 try 0 launched on executor multi with executor id 2286 parsl.dataflow.dflow INFO: Standard output for task 2285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3d1e8c57-56c5-4741-838e-1dde65a69921_isrForCrosstalkSources_1125912_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3d1e8c57-56c5-4741-838e-1dde65a69921_isrForCrosstalkSources_1125912_10.stderr parsl.dataflow.dflow DEBUG: Task 2286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2286 has memoization hash f46aa2c7cbe40220dbc7da741662d8f3 parsl.dataflow.memoization INFO: Task 2286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2286 try 0 launched on executor multi with executor id 2287 parsl.dataflow.dflow INFO: Standard output for task 2286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/6de77a45-14af-4df0-861e-dbc58d28a3a0_isrForCrosstalkSources_1125930_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/6de77a45-14af-4df0-861e-dbc58d28a3a0_isrForCrosstalkSources_1125930_31.stderr parsl.dataflow.dflow DEBUG: Task 2287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2287 has memoization hash de2d98d971466bc7e212c37d4e6705f0 parsl.dataflow.memoization INFO: Task 2287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2287 try 0 launched on executor multi with executor id 2288 parsl.dataflow.dflow INFO: Standard output for task 2287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f9914589-add5-4e71-ad98-ef3ac5cd9f75_isrForCrosstalkSources_1125945_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f9914589-add5-4e71-ad98-ef3ac5cd9f75_isrForCrosstalkSources_1125945_8.stderr parsl.dataflow.dflow DEBUG: Task 2288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2288 has memoization hash 632749d222163a826de807b8aa9c4bb5 parsl.dataflow.memoization INFO: Task 2288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397498a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2288 try 0 launched on executor multi with executor id 2289 parsl.dataflow.dflow INFO: Standard output for task 2288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/23e8b9fa-0dbe-421d-b044-8b9973c8224b_isrForCrosstalkSources_1125916_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/23e8b9fa-0dbe-421d-b044-8b9973c8224b_isrForCrosstalkSources_1125916_6.stderr parsl.dataflow.dflow DEBUG: Task 2289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2289 has memoization hash 582cd82deb6d68e1412e4b62e19ffb94 parsl.dataflow.memoization INFO: Task 2289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397499e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2289 try 0 launched on executor multi with executor id 2290 parsl.dataflow.dflow INFO: Standard output for task 2289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/618505d0-228f-4db3-b33c-11f9cf26058c_isrForCrosstalkSources_1125927_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/618505d0-228f-4db3-b33c-11f9cf26058c_isrForCrosstalkSources_1125927_43.stderr parsl.dataflow.dflow DEBUG: Task 2290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2290 has memoization hash 211bdda86ab39e104576a681491fc830 parsl.dataflow.memoization INFO: Task 2290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2290 try 0 launched on executor multi with executor id 2291 parsl.dataflow.dflow INFO: Standard output for task 2290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/b16391ed-d4b2-4c79-9ab6-6d5d241c8f49_isrForCrosstalkSources_1125930_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/b16391ed-d4b2-4c79-9ab6-6d5d241c8f49_isrForCrosstalkSources_1125930_26.stderr parsl.dataflow.dflow DEBUG: Task 2291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2291 has memoization hash 4b013285596e2896bb15ce2c3dec19fa parsl.dataflow.memoization INFO: Task 2291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2291 try 0 launched on executor multi with executor id 2292 parsl.dataflow.dflow INFO: Standard output for task 2291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/8fe1660f-5135-44d4-99dd-430a71a88c7e_isrForCrosstalkSources_1125935_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/8fe1660f-5135-44d4-99dd-430a71a88c7e_isrForCrosstalkSources_1125935_42.stderr parsl.dataflow.dflow DEBUG: Task 2292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2292 has memoization hash cdc317df28589c1bf41adb297043f63a parsl.dataflow.memoization INFO: Task 2292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2292 try 0 launched on executor multi with executor id 2293 parsl.dataflow.dflow INFO: Standard output for task 2292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/f92f1455-22b5-40c8-a812-3b179cfa8cc4_isrForCrosstalkSources_1125929_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/f92f1455-22b5-40c8-a812-3b179cfa8cc4_isrForCrosstalkSources_1125929_26.stderr parsl.dataflow.dflow DEBUG: Task 2293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2293 has memoization hash a21207fef6792dbbf088c5f1d0720909 parsl.dataflow.memoization INFO: Task 2293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2293 try 0 launched on executor multi with executor id 2294 parsl.dataflow.dflow INFO: Standard output for task 2293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/30c456a6-816a-496b-a700-209e7eae47df_isrForCrosstalkSources_1125907_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/30c456a6-816a-496b-a700-209e7eae47df_isrForCrosstalkSources_1125907_18.stderr parsl.dataflow.dflow DEBUG: Task 2294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2294 has memoization hash b33e8f0f2fca07e4df49c43cb0050d9f parsl.dataflow.memoization INFO: Task 2294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2294 try 0 launched on executor multi with executor id 2295 parsl.dataflow.dflow INFO: Standard output for task 2294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/80daaa2f-1043-4424-8775-4a3856479206_isrForCrosstalkSources_1125930_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/80daaa2f-1043-4424-8775-4a3856479206_isrForCrosstalkSources_1125930_14.stderr parsl.dataflow.dflow DEBUG: Task 2295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2295 has memoization hash 112ce67af24992d1c6e9bb18f8440364 parsl.dataflow.memoization INFO: Task 2295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2295 try 0 launched on executor multi with executor id 2296 parsl.dataflow.dflow INFO: Standard output for task 2295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/1840d645-6d91-4d7b-b9fc-eb8867866769_isrForCrosstalkSources_1125917_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/1840d645-6d91-4d7b-b9fc-eb8867866769_isrForCrosstalkSources_1125917_51.stderr parsl.dataflow.dflow DEBUG: Task 2296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2296 has memoization hash 37887d36ad04d7fa7cec6a1b7b39da66 parsl.dataflow.memoization INFO: Task 2296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2296 try 0 launched on executor multi with executor id 2297 parsl.dataflow.dflow INFO: Standard output for task 2296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/45d047f1-19a8-4a7b-9d90-e9ff55bb1f5f_isrForCrosstalkSources_1125939_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/45d047f1-19a8-4a7b-9d90-e9ff55bb1f5f_isrForCrosstalkSources_1125939_46.stderr parsl.dataflow.dflow DEBUG: Task 2297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2297 has memoization hash 598086ebddc98da5523b29f8ce664361 parsl.dataflow.memoization INFO: Task 2297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2297 try 0 launched on executor multi with executor id 2298 parsl.dataflow.dflow INFO: Standard output for task 2297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/4a0ca294-1a79-4096-80fc-dd9cacacee8d_isrForCrosstalkSources_1125929_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/4a0ca294-1a79-4096-80fc-dd9cacacee8d_isrForCrosstalkSources_1125929_32.stderr parsl.dataflow.dflow DEBUG: Task 2298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2298 has memoization hash 8897b94472d92c5c9b392bf20430a849 parsl.dataflow.memoization INFO: Task 2298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39749ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2298 try 0 launched on executor multi with executor id 2299 parsl.dataflow.dflow INFO: Standard output for task 2298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/bc5e32d3-2bbf-484c-be60-c4fc25cf70be_isrForCrosstalkSources_1125891_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/bc5e32d3-2bbf-484c-be60-c4fc25cf70be_isrForCrosstalkSources_1125891_21.stderr parsl.dataflow.dflow DEBUG: Task 2299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2299 has memoization hash 24b4c8f4093a159b975314e90f17c0da parsl.dataflow.memoization INFO: Task 2299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2299 try 0 launched on executor multi with executor id 2300 parsl.dataflow.dflow INFO: Standard output for task 2299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/47a7def7-30e7-4014-839b-97b04201129f_isrForCrosstalkSources_1125914_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/47a7def7-30e7-4014-839b-97b04201129f_isrForCrosstalkSources_1125914_40.stderr parsl.dataflow.dflow DEBUG: Task 2300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2300 has memoization hash 138e962192bab77f381aafeac430b312 parsl.dataflow.memoization INFO: Task 2300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2300 try 0 launched on executor multi with executor id 2301 parsl.dataflow.dflow INFO: Standard output for task 2300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/ad3d6ece-a558-47c5-a4ee-848e39977502_isrForCrosstalkSources_1125933_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/ad3d6ece-a558-47c5-a4ee-848e39977502_isrForCrosstalkSources_1125933_33.stderr parsl.dataflow.dflow DEBUG: Task 2301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2301 has memoization hash 26e6de5ce77f40c2cd1bf04e70da0879 parsl.dataflow.memoization INFO: Task 2301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2301 try 0 launched on executor multi with executor id 2302 parsl.dataflow.dflow INFO: Standard output for task 2301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/74706dbc-797e-4173-b714-2bde268b47a6_isrForCrosstalkSources_1125894_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/74706dbc-797e-4173-b714-2bde268b47a6_isrForCrosstalkSources_1125894_59.stderr parsl.dataflow.dflow DEBUG: Task 2302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2302 has memoization hash 6d888608031bc6283b335ca9d59c5e8d parsl.dataflow.memoization INFO: Task 2302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2302 try 0 launched on executor multi with executor id 2303 parsl.dataflow.dflow INFO: Standard output for task 2302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/f541d420-fb63-428a-800e-79138f31bed0_isrForCrosstalkSources_1125916_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/f541d420-fb63-428a-800e-79138f31bed0_isrForCrosstalkSources_1125916_41.stderr parsl.dataflow.dflow DEBUG: Task 2303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2303 has memoization hash 598730f078ac1274ab514cb8fad117f4 parsl.dataflow.memoization INFO: Task 2303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2303 try 0 launched on executor multi with executor id 2304 parsl.dataflow.dflow INFO: Standard output for task 2303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/e23ef477-f5be-4b24-b12f-1aee4197da03_isrForCrosstalkSources_1125893_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/e23ef477-f5be-4b24-b12f-1aee4197da03_isrForCrosstalkSources_1125893_12.stderr parsl.dataflow.dflow DEBUG: Task 2304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2304 has memoization hash cebe5a84a0354b013080b0d53b065365 parsl.dataflow.memoization INFO: Task 2304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2304 try 0 launched on executor multi with executor id 2305 parsl.dataflow.dflow INFO: Standard output for task 2304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/a3bb0ce2-00f8-466f-bdb8-57710a757fbe_isrForCrosstalkSources_1125908_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/a3bb0ce2-00f8-466f-bdb8-57710a757fbe_isrForCrosstalkSources_1125908_42.stderr parsl.dataflow.dflow DEBUG: Task 2305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2305 has memoization hash ad4f2f83b37dcf032d5cf93e21fb1e0b parsl.dataflow.memoization INFO: Task 2305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2305 try 0 launched on executor multi with executor id 2306 parsl.dataflow.dflow INFO: Standard output for task 2305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/051867cb-935a-467a-81d2-d1021c011f92_isrForCrosstalkSources_1125932_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/051867cb-935a-467a-81d2-d1021c011f92_isrForCrosstalkSources_1125932_37.stderr parsl.dataflow.dflow DEBUG: Task 2306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2306 has memoization hash c12a42ba6ba25a7c3cd9c113a31bba14 parsl.dataflow.memoization INFO: Task 2306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2306 try 0 launched on executor multi with executor id 2307 parsl.dataflow.dflow INFO: Standard output for task 2306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/d35d2e45-3781-4361-97c8-4776afac9826_isrForCrosstalkSources_1125928_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/d35d2e45-3781-4361-97c8-4776afac9826_isrForCrosstalkSources_1125928_48.stderr parsl.dataflow.dflow DEBUG: Task 2307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2307 has memoization hash 58be8aa93a85aacf97d9d05c7394df0f parsl.dataflow.memoization INFO: Task 2307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2307 try 0 launched on executor multi with executor id 2308 parsl.dataflow.dflow INFO: Standard output for task 2307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/02ddcd9f-32d7-41c5-8c07-104af8d3c53c_isrForCrosstalkSources_1125945_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/02ddcd9f-32d7-41c5-8c07-104af8d3c53c_isrForCrosstalkSources_1125945_62.stderr parsl.dataflow.dflow DEBUG: Task 2308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2308 has memoization hash d552b6e63d7f9e5cac214382ce2efa74 parsl.dataflow.memoization INFO: Task 2308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2308 try 0 launched on executor multi with executor id 2309 parsl.dataflow.dflow INFO: Standard output for task 2308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/550d89f0-486e-4b3c-9f73-427a6d5bb970_isrForCrosstalkSources_1125944_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/550d89f0-486e-4b3c-9f73-427a6d5bb970_isrForCrosstalkSources_1125944_14.stderr parsl.dataflow.dflow DEBUG: Task 2309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2309 has memoization hash 1266094524b2e18e0b7528f2dc2a16a3 parsl.dataflow.memoization INFO: Task 2309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2309 try 0 launched on executor multi with executor id 2310 parsl.dataflow.dflow INFO: Standard output for task 2309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/7cc750a4-830d-40a2-9478-a1cc8f7a4e37_isrForCrosstalkSources_1125910_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/7cc750a4-830d-40a2-9478-a1cc8f7a4e37_isrForCrosstalkSources_1125910_40.stderr parsl.dataflow.dflow DEBUG: Task 2310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2310 has memoization hash fbb400e309b1ef8ab5f9feedad1bf18f parsl.dataflow.memoization INFO: Task 2310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2310 try 0 launched on executor multi with executor id 2311 parsl.dataflow.dflow INFO: Standard output for task 2310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/98f9cf9e-2db5-4965-8abe-6977f3ffa81a_isrForCrosstalkSources_1125896_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/98f9cf9e-2db5-4965-8abe-6977f3ffa81a_isrForCrosstalkSources_1125896_29.stderr parsl.dataflow.dflow DEBUG: Task 2311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2311 has memoization hash 7a10e07311c2ca00fe2c316beeb6d9d8 parsl.dataflow.memoization INFO: Task 2311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2311 try 0 launched on executor multi with executor id 2312 parsl.dataflow.dflow INFO: Standard output for task 2311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/c9fb20c1-3f45-4c80-9907-ee0fb2788f3e_isrForCrosstalkSources_1125907_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/c9fb20c1-3f45-4c80-9907-ee0fb2788f3e_isrForCrosstalkSources_1125907_59.stderr parsl.dataflow.dflow DEBUG: Task 2312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2312 has memoization hash c037b6093bcce4bbba15a94d4d4e3e6a parsl.dataflow.memoization INFO: Task 2312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2312 try 0 launched on executor multi with executor id 2313 parsl.dataflow.dflow INFO: Standard output for task 2312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/707d562f-c4ae-4f2a-bd9a-e0214f1afa04_isrForCrosstalkSources_1125899_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/707d562f-c4ae-4f2a-bd9a-e0214f1afa04_isrForCrosstalkSources_1125899_54.stderr parsl.dataflow.dflow DEBUG: Task 2313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2313 has memoization hash 975b0da3600fd79552389a53029ba7ab parsl.dataflow.memoization INFO: Task 2313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2313 try 0 launched on executor multi with executor id 2314 parsl.dataflow.dflow INFO: Standard output for task 2313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/5698ddf9-fa2a-4cd5-accd-fe9256e48cb4_isrForCrosstalkSources_1125936_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/5698ddf9-fa2a-4cd5-accd-fe9256e48cb4_isrForCrosstalkSources_1125936_29.stderr parsl.dataflow.dflow DEBUG: Task 2314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2314 has memoization hash 3c642101ebb16d17a4aa33fb72490477 parsl.dataflow.memoization INFO: Task 2314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2314 try 0 launched on executor multi with executor id 2315 parsl.dataflow.dflow INFO: Standard output for task 2314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/26eae107-3227-4056-9775-62ef50a710bb_isrForCrosstalkSources_1125896_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/26eae107-3227-4056-9775-62ef50a710bb_isrForCrosstalkSources_1125896_26.stderr parsl.dataflow.dflow DEBUG: Task 2315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2315 has memoization hash e8f9567c8f9f26af1f85b14b571c5d2c parsl.dataflow.memoization INFO: Task 2315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2315 try 0 launched on executor multi with executor id 2316 parsl.dataflow.dflow INFO: Standard output for task 2315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/ca1b7f6a-6ad7-43d6-9536-3f0e52d0f494_isrForCrosstalkSources_1125937_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/ca1b7f6a-6ad7-43d6-9536-3f0e52d0f494_isrForCrosstalkSources_1125937_13.stderr parsl.dataflow.dflow DEBUG: Task 2316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2316 has memoization hash b1a6e9ca7636ea326c8bc5ef0c475c88 parsl.dataflow.memoization INFO: Task 2316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2316 try 0 launched on executor multi with executor id 2317 parsl.dataflow.dflow INFO: Standard output for task 2316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/036a1450-0b25-4f8f-9a39-94b312cb3700_isrForCrosstalkSources_1125933_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/036a1450-0b25-4f8f-9a39-94b312cb3700_isrForCrosstalkSources_1125933_19.stderr parsl.dataflow.dflow DEBUG: Task 2317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2317 has memoization hash 22c9810cb37d1f3fb9f6dd47c8e3f581 parsl.dataflow.memoization INFO: Task 2317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2317 try 0 launched on executor multi with executor id 2318 parsl.dataflow.dflow INFO: Standard output for task 2317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/95df473d-7618-41b3-af61-6c713950c5bf_isrForCrosstalkSources_1125912_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/95df473d-7618-41b3-af61-6c713950c5bf_isrForCrosstalkSources_1125912_32.stderr parsl.dataflow.dflow DEBUG: Task 2318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2318 has memoization hash 91b87210069f399ae5e4773242be4847 parsl.dataflow.memoization INFO: Task 2318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2318 try 0 launched on executor multi with executor id 2319 parsl.dataflow.dflow INFO: Standard output for task 2318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/4e8744e5-2775-47e5-b90d-1df7d0fbb441_isrForCrosstalkSources_1125890_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/4e8744e5-2775-47e5-b90d-1df7d0fbb441_isrForCrosstalkSources_1125890_6.stderr parsl.dataflow.dflow DEBUG: Task 2319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2319 has memoization hash 8bcfa20f9d4786eb4477602ca44f4c29 parsl.dataflow.memoization INFO: Task 2319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2319 try 0 launched on executor multi with executor id 2320 parsl.dataflow.dflow INFO: Standard output for task 2319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/ecadd54d-f3ba-4ef8-a02a-1ae811ac9b83_isrForCrosstalkSources_1125891_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/ecadd54d-f3ba-4ef8-a02a-1ae811ac9b83_isrForCrosstalkSources_1125891_35.stderr parsl.dataflow.dflow DEBUG: Task 2320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2320 has memoization hash ea475d162da7f4197197bd152ffd862d parsl.dataflow.memoization INFO: Task 2320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2320 try 0 launched on executor multi with executor id 2321 parsl.dataflow.dflow INFO: Standard output for task 2320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/ee4233d2-933b-4ccb-b001-313e28f26ceb_isrForCrosstalkSources_1125890_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/ee4233d2-933b-4ccb-b001-313e28f26ceb_isrForCrosstalkSources_1125890_19.stderr parsl.dataflow.dflow DEBUG: Task 2321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2321 has memoization hash c13f884dab711e62d996afc8b8c33e4c parsl.dataflow.memoization INFO: Task 2321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2321 try 0 launched on executor multi with executor id 2322 parsl.dataflow.dflow INFO: Standard output for task 2321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/0af0a581-5828-433a-8e6b-d3f938eed9ac_isrForCrosstalkSources_1125894_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/0af0a581-5828-433a-8e6b-d3f938eed9ac_isrForCrosstalkSources_1125894_48.stderr parsl.dataflow.dflow DEBUG: Task 2322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2322 has memoization hash 2d937b1c298add24cf7e55634f3e117f parsl.dataflow.memoization INFO: Task 2322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2322 try 0 launched on executor multi with executor id 2323 parsl.dataflow.dflow INFO: Standard output for task 2322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/b307b780-7483-4481-8236-1c666f314f4e_isrForCrosstalkSources_1125892_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/b307b780-7483-4481-8236-1c666f314f4e_isrForCrosstalkSources_1125892_28.stderr parsl.dataflow.dflow DEBUG: Task 2323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2323 has memoization hash 11a5fd2280be5a92faba621341a8b6f2 parsl.dataflow.memoization INFO: Task 2323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2323 try 0 launched on executor multi with executor id 2324 parsl.dataflow.dflow INFO: Standard output for task 2323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/96c6667f-be39-4bb8-b585-a6fbfbbaf5a0_isrForCrosstalkSources_1125941_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/96c6667f-be39-4bb8-b585-a6fbfbbaf5a0_isrForCrosstalkSources_1125941_7.stderr parsl.dataflow.dflow DEBUG: Task 2324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2324 has memoization hash 8302e894f779450cdc807bbdf22f79f5 parsl.dataflow.memoization INFO: Task 2324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2324 try 0 launched on executor multi with executor id 2325 parsl.dataflow.dflow INFO: Standard output for task 2324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/763eef04-181d-40b4-a784-feecb3efcdca_isrForCrosstalkSources_1125943_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/763eef04-181d-40b4-a784-feecb3efcdca_isrForCrosstalkSources_1125943_41.stderr parsl.dataflow.dflow DEBUG: Task 2325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2325 has memoization hash cc8a1177a7c40cf9154aa937d97751af parsl.dataflow.memoization INFO: Task 2325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2325 try 0 launched on executor multi with executor id 2326 parsl.dataflow.dflow INFO: Standard output for task 2325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/723dd616-72a4-434a-bcf3-1aed555bf91a_isrForCrosstalkSources_1125914_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/723dd616-72a4-434a-bcf3-1aed555bf91a_isrForCrosstalkSources_1125914_34.stderr parsl.dataflow.dflow DEBUG: Task 2326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2326 has memoization hash 3d99917bde3b8b15e03db047a53baeb1 parsl.dataflow.memoization INFO: Task 2326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2326 try 0 launched on executor multi with executor id 2327 parsl.dataflow.dflow INFO: Standard output for task 2326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/1d8cc44a-5e3e-4719-b064-e88556a49abd_isrForCrosstalkSources_1125917_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/1d8cc44a-5e3e-4719-b064-e88556a49abd_isrForCrosstalkSources_1125917_52.stderr parsl.dataflow.dflow DEBUG: Task 2327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2327 has memoization hash 10efe3fe124d7fd44ba5ae43a78bbcff parsl.dataflow.memoization INFO: Task 2327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2327 try 0 launched on executor multi with executor id 2328 parsl.dataflow.dflow INFO: Standard output for task 2327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/5bf7d879-7051-4c77-8d74-6d12ef6eefc0_isrForCrosstalkSources_1125934_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/5bf7d879-7051-4c77-8d74-6d12ef6eefc0_isrForCrosstalkSources_1125934_12.stderr parsl.dataflow.dflow DEBUG: Task 2328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2328 has memoization hash 7f38a5e04752dedb4b7ffdeccbeea03d parsl.dataflow.memoization INFO: Task 2328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2328 try 0 launched on executor multi with executor id 2329 parsl.dataflow.dflow INFO: Standard output for task 2328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/a9c9b639-e2df-43b9-8514-08b06e62d5a0_isrForCrosstalkSources_1125940_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/a9c9b639-e2df-43b9-8514-08b06e62d5a0_isrForCrosstalkSources_1125940_47.stderr parsl.dataflow.dflow DEBUG: Task 2329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2329 has memoization hash 4407086b1757bfa5056f05a06abd9c6d parsl.dataflow.memoization INFO: Task 2329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2329 try 0 launched on executor multi with executor id 2330 parsl.dataflow.dflow INFO: Standard output for task 2329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/0c1af82e-4d40-435b-ba52-6860027978b6_isrForCrosstalkSources_1125935_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/0c1af82e-4d40-435b-ba52-6860027978b6_isrForCrosstalkSources_1125935_3.stderr parsl.dataflow.dflow DEBUG: Task 2330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2330 has memoization hash 3d9f94ab6f15939269c4eeb3a2c5681f parsl.dataflow.memoization INFO: Task 2330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2330 try 0 launched on executor multi with executor id 2331 parsl.dataflow.dflow INFO: Standard output for task 2330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/6324ee1e-8fd4-4bd5-8d89-b9ecbd39f4d3_isrForCrosstalkSources_1125892_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/6324ee1e-8fd4-4bd5-8d89-b9ecbd39f4d3_isrForCrosstalkSources_1125892_25.stderr parsl.dataflow.dflow DEBUG: Task 2331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2331 has memoization hash 5f6978e198ecfd4ce6d65007a87e3e62 parsl.dataflow.memoization INFO: Task 2331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2331 try 0 launched on executor multi with executor id 2332 parsl.dataflow.dflow INFO: Standard output for task 2331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/50937c70-402f-4143-a99e-2c4ec3a26969_isrForCrosstalkSources_1125909_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/50937c70-402f-4143-a99e-2c4ec3a26969_isrForCrosstalkSources_1125909_51.stderr parsl.dataflow.dflow DEBUG: Task 2332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2332 has memoization hash ccb6ce120800c911dfb570cea3989c7b parsl.dataflow.memoization INFO: Task 2332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2332 try 0 launched on executor multi with executor id 2333 parsl.dataflow.dflow INFO: Standard output for task 2332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/c84a730a-75d2-4019-99f1-fcf42cde329d_isrForCrosstalkSources_1125920_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/c84a730a-75d2-4019-99f1-fcf42cde329d_isrForCrosstalkSources_1125920_31.stderr parsl.dataflow.dflow DEBUG: Task 2333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2333 has memoization hash ad6885fa3e33fb63c1d976a8a8a88318 parsl.dataflow.memoization INFO: Task 2333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2333 try 0 launched on executor multi with executor id 2334 parsl.dataflow.dflow INFO: Standard output for task 2333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/e388a90c-51a9-4960-8f62-3445337f285f_isrForCrosstalkSources_1125902_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/e388a90c-51a9-4960-8f62-3445337f285f_isrForCrosstalkSources_1125902_6.stderr parsl.dataflow.dflow DEBUG: Task 2334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2334 has memoization hash c450c2d36a77fe4bb06444b2c3d08a1d parsl.dataflow.memoization INFO: Task 2334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2334 try 0 launched on executor multi with executor id 2335 parsl.dataflow.dflow INFO: Standard output for task 2334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/754b65a4-098e-4e2f-a784-f1e7c1e8a203_isrForCrosstalkSources_1125904_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/754b65a4-098e-4e2f-a784-f1e7c1e8a203_isrForCrosstalkSources_1125904_28.stderr parsl.dataflow.dflow DEBUG: Task 2335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2335 has memoization hash f83ab7395e980fa4b544079b0036a1f8 parsl.dataflow.memoization INFO: Task 2335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2335 try 0 launched on executor multi with executor id 2336 parsl.dataflow.dflow INFO: Standard output for task 2335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/e2803dfc-c1be-4c6f-a3d7-8e95737a6fe9_isrForCrosstalkSources_1125909_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/e2803dfc-c1be-4c6f-a3d7-8e95737a6fe9_isrForCrosstalkSources_1125909_28.stderr parsl.dataflow.dflow DEBUG: Task 2336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2336 has memoization hash edd9406ec829b400c4def7f55258f943 parsl.dataflow.memoization INFO: Task 2336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2336 try 0 launched on executor multi with executor id 2337 parsl.dataflow.dflow INFO: Standard output for task 2336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/5b74e65a-b362-488e-9ce8-69d21d76cf12_isrForCrosstalkSources_1125889_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/5b74e65a-b362-488e-9ce8-69d21d76cf12_isrForCrosstalkSources_1125889_24.stderr parsl.dataflow.dflow DEBUG: Task 2337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2337 has memoization hash f2d3b530d5f2ec66cb8dbed0422a923f parsl.dataflow.memoization INFO: Task 2337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2337 try 0 launched on executor multi with executor id 2338 parsl.dataflow.dflow INFO: Standard output for task 2337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/b752884d-d521-472c-9e8d-add75e8e9a0e_isrForCrosstalkSources_1125945_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/b752884d-d521-472c-9e8d-add75e8e9a0e_isrForCrosstalkSources_1125945_20.stderr parsl.dataflow.dflow DEBUG: Task 2338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2338 has memoization hash 4d26effb6c16717f1188a1035a7d6051 parsl.dataflow.memoization INFO: Task 2338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2338 try 0 launched on executor multi with executor id 2339 parsl.dataflow.dflow INFO: Standard output for task 2338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/c15ab890-5984-4b13-816e-f013c40afad6_isrForCrosstalkSources_1125899_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/c15ab890-5984-4b13-816e-f013c40afad6_isrForCrosstalkSources_1125899_24.stderr parsl.dataflow.dflow DEBUG: Task 2339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2339 has memoization hash ecb7bc4f94edf2700c72bc2daa6a1c5d parsl.dataflow.memoization INFO: Task 2339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2339 try 0 launched on executor multi with executor id 2340 parsl.dataflow.dflow INFO: Standard output for task 2339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/7748ccb5-5ac8-4f92-bb31-54cc18cef6f1_isrForCrosstalkSources_1125907_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/7748ccb5-5ac8-4f92-bb31-54cc18cef6f1_isrForCrosstalkSources_1125907_6.stderr parsl.dataflow.dflow DEBUG: Task 2340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2340 has memoization hash 2482a27c8b6496d0db8bbf43851e35f5 parsl.dataflow.memoization INFO: Task 2340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2340 try 0 launched on executor multi with executor id 2341 parsl.dataflow.dflow INFO: Standard output for task 2340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/8136dcb2-46b3-4a3a-957b-dc56265901b0_isrForCrosstalkSources_1125934_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/8136dcb2-46b3-4a3a-957b-dc56265901b0_isrForCrosstalkSources_1125934_20.stderr parsl.dataflow.dflow DEBUG: Task 2341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2341 has memoization hash 0fcecdfe7d10f834aac31a73255ec571 parsl.dataflow.memoization INFO: Task 2341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2341 try 0 launched on executor multi with executor id 2342 parsl.dataflow.dflow INFO: Standard output for task 2341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/11afdd0a-9236-49d0-aaba-130362e42526_isrForCrosstalkSources_1125917_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/11afdd0a-9236-49d0-aaba-130362e42526_isrForCrosstalkSources_1125917_58.stderr parsl.dataflow.dflow DEBUG: Task 2342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2342 has memoization hash 3de46d899c0bdee0deed13a3637bf163 parsl.dataflow.memoization INFO: Task 2342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2342 try 0 launched on executor multi with executor id 2343 parsl.dataflow.dflow INFO: Standard output for task 2342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/188cdf08-3d53-49a9-b7a1-37fba0d6465d_isrForCrosstalkSources_1125925_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/188cdf08-3d53-49a9-b7a1-37fba0d6465d_isrForCrosstalkSources_1125925_32.stderr parsl.dataflow.dflow DEBUG: Task 2343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2343 has memoization hash 19cac0c4046829c93743a144612cfbf3 parsl.dataflow.memoization INFO: Task 2343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2343 try 0 launched on executor multi with executor id 2344 parsl.dataflow.dflow INFO: Standard output for task 2343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/21e66ce4-dcc1-4b39-92ed-1112fb50d9dc_isrForCrosstalkSources_1125913_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/21e66ce4-dcc1-4b39-92ed-1112fb50d9dc_isrForCrosstalkSources_1125913_8.stderr parsl.dataflow.dflow DEBUG: Task 2344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2344 has memoization hash fc74dcaf3e79b614c72d55c6c048d467 parsl.dataflow.memoization INFO: Task 2344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2344 try 0 launched on executor multi with executor id 2345 parsl.dataflow.dflow INFO: Standard output for task 2344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/05724e60-bcfe-461a-84c1-ecb86def82da_isrForCrosstalkSources_1125940_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/05724e60-bcfe-461a-84c1-ecb86def82da_isrForCrosstalkSources_1125940_50.stderr parsl.dataflow.dflow DEBUG: Task 2345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2345 has memoization hash 720f9e4650092d24b2bc69e232325071 parsl.dataflow.memoization INFO: Task 2345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2345 try 0 launched on executor multi with executor id 2346 parsl.dataflow.dflow INFO: Standard output for task 2345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e1ca6050-78c4-43f4-8b06-d5bcdaae93da_isrForCrosstalkSources_1125901_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e1ca6050-78c4-43f4-8b06-d5bcdaae93da_isrForCrosstalkSources_1125901_24.stderr parsl.dataflow.dflow DEBUG: Task 2346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2346 has memoization hash b917a437a8ce9d4e5daba3ea85dcc293 parsl.dataflow.memoization INFO: Task 2346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2346 try 0 launched on executor multi with executor id 2347 parsl.dataflow.dflow INFO: Standard output for task 2346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/be5004ee-1740-452a-9a41-840ecfa42a35_isrForCrosstalkSources_1125891_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/be5004ee-1740-452a-9a41-840ecfa42a35_isrForCrosstalkSources_1125891_62.stderr parsl.dataflow.dflow DEBUG: Task 2347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2347 has memoization hash 941ff3e0a814acc58603d43b79268aad parsl.dataflow.memoization INFO: Task 2347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2347 try 0 launched on executor multi with executor id 2348 parsl.dataflow.dflow INFO: Standard output for task 2347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/65e723b5-1a5c-4906-8981-c2ae9c440b6c_isrForCrosstalkSources_1125897_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/65e723b5-1a5c-4906-8981-c2ae9c440b6c_isrForCrosstalkSources_1125897_10.stderr parsl.dataflow.dflow DEBUG: Task 2348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2348 has memoization hash ab63b077a292b4fe9a67780b0736ec55 parsl.dataflow.memoization INFO: Task 2348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2348 try 0 launched on executor multi with executor id 2349 parsl.dataflow.dflow INFO: Standard output for task 2348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/6e8bcc71-c2bf-44a0-ba70-71e99858ce9a_isrForCrosstalkSources_1125945_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/6e8bcc71-c2bf-44a0-ba70-71e99858ce9a_isrForCrosstalkSources_1125945_2.stderr parsl.dataflow.dflow DEBUG: Task 2349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2349 has memoization hash 2e2748386210171055ab2f059bdeccd0 parsl.dataflow.memoization INFO: Task 2349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3974bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2349 try 0 launched on executor multi with executor id 2350 parsl.dataflow.dflow INFO: Standard output for task 2349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/314c016e-6af3-497b-8c47-5f0175cd47d1_isrForCrosstalkSources_1125916_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/314c016e-6af3-497b-8c47-5f0175cd47d1_isrForCrosstalkSources_1125916_16.stderr parsl.dataflow.dflow DEBUG: Task 2350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2350 has memoization hash cb736f88c87b2606532adf473db0c7d2 parsl.dataflow.memoization INFO: Task 2350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39748040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2350 try 0 launched on executor multi with executor id 2351 parsl.dataflow.dflow INFO: Standard output for task 2350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/7ad9f680-e74b-43c9-803c-b9e22f5d1553_isrForCrosstalkSources_1125902_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/7ad9f680-e74b-43c9-803c-b9e22f5d1553_isrForCrosstalkSources_1125902_14.stderr parsl.dataflow.dflow DEBUG: Task 2351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2351 has memoization hash 5e0a0d3201fe9ed75f4e3a846a01a689 parsl.dataflow.memoization INFO: Task 2351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2351 try 0 launched on executor multi with executor id 2352 parsl.dataflow.dflow INFO: Standard output for task 2351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/bff1be35-7859-470f-9ee6-54e06f95fb12_isrForCrosstalkSources_1125923_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/bff1be35-7859-470f-9ee6-54e06f95fb12_isrForCrosstalkSources_1125923_20.stderr parsl.dataflow.dflow DEBUG: Task 2352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2352 has memoization hash 0f93ff64f9aa71c056b96cece69d6735 parsl.dataflow.memoization INFO: Task 2352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2352 try 0 launched on executor multi with executor id 2353 parsl.dataflow.dflow INFO: Standard output for task 2352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/4f00b3b4-c73c-4c93-a84a-c00813956f0c_isrForCrosstalkSources_1125897_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/4f00b3b4-c73c-4c93-a84a-c00813956f0c_isrForCrosstalkSources_1125897_40.stderr parsl.dataflow.dflow DEBUG: Task 2353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2353 has memoization hash 2ea90ab3603fc668b6d18878dffe6d99 parsl.dataflow.memoization INFO: Task 2353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2353 try 0 launched on executor multi with executor id 2354 parsl.dataflow.dflow INFO: Standard output for task 2353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/36c56643-b097-4c1c-8981-9f4846d572cc_isrForCrosstalkSources_1125891_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/36c56643-b097-4c1c-8981-9f4846d572cc_isrForCrosstalkSources_1125891_7.stderr parsl.dataflow.dflow DEBUG: Task 2354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2354 has memoization hash 57ef7382f448aa1fe911f079f4521dc8 parsl.dataflow.memoization INFO: Task 2354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2354 try 0 launched on executor multi with executor id 2355 parsl.dataflow.dflow INFO: Standard output for task 2354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/e83b3edf-346b-42d3-b0d2-f6ee8f469ce4_isrForCrosstalkSources_1125907_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/e83b3edf-346b-42d3-b0d2-f6ee8f469ce4_isrForCrosstalkSources_1125907_17.stderr parsl.dataflow.dflow DEBUG: Task 2355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2355 has memoization hash 06d371ad046be6dfafeeb6442babfa49 parsl.dataflow.memoization INFO: Task 2355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2355 try 0 launched on executor multi with executor id 2356 parsl.dataflow.dflow INFO: Standard output for task 2355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/cc617dbe-cc0c-4f93-bb0c-199bf437f6f4_isrForCrosstalkSources_1125941_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/cc617dbe-cc0c-4f93-bb0c-199bf437f6f4_isrForCrosstalkSources_1125941_59.stderr parsl.dataflow.dflow DEBUG: Task 2356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2356 has memoization hash 83aa6f62d38baccd77b81c43cfcb3d57 parsl.dataflow.memoization INFO: Task 2356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2356 try 0 launched on executor multi with executor id 2357 parsl.dataflow.dflow INFO: Standard output for task 2356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/ac3d1199-1eb0-4179-8406-b06a0d9ac7cf_isrForCrosstalkSources_1125908_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/ac3d1199-1eb0-4179-8406-b06a0d9ac7cf_isrForCrosstalkSources_1125908_23.stderr parsl.dataflow.dflow DEBUG: Task 2357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2357 has memoization hash f2ee181553cf1056b968713733368587 parsl.dataflow.memoization INFO: Task 2357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2357 try 0 launched on executor multi with executor id 2358 parsl.dataflow.dflow INFO: Standard output for task 2357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/dfbe6b3f-cd26-4c06-9523-0814f14cf361_isrForCrosstalkSources_1125942_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/dfbe6b3f-cd26-4c06-9523-0814f14cf361_isrForCrosstalkSources_1125942_21.stderr parsl.dataflow.dflow DEBUG: Task 2358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2358 has memoization hash c3cbb908a62ef5ca6e17356b86c976cf parsl.dataflow.memoization INFO: Task 2358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2358 try 0 launched on executor multi with executor id 2359 parsl.dataflow.dflow INFO: Standard output for task 2358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/2cd9d7de-7b0e-4326-9b50-85be44407a8d_isrForCrosstalkSources_1125942_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/2cd9d7de-7b0e-4326-9b50-85be44407a8d_isrForCrosstalkSources_1125942_45.stderr parsl.dataflow.dflow DEBUG: Task 2359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2359 has memoization hash c5d7f5f63781904e275679c8f0c61023 parsl.dataflow.memoization INFO: Task 2359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2359 try 0 launched on executor multi with executor id 2360 parsl.dataflow.dflow INFO: Standard output for task 2359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/09d9474e-db23-438f-91c8-06ec2ff43750_isrForCrosstalkSources_1125937_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/09d9474e-db23-438f-91c8-06ec2ff43750_isrForCrosstalkSources_1125937_52.stderr parsl.dataflow.dflow DEBUG: Task 2360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2360 has memoization hash 3d23035436028266565a314db67c0ed6 parsl.dataflow.memoization INFO: Task 2360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2360 try 0 launched on executor multi with executor id 2361 parsl.dataflow.dflow INFO: Standard output for task 2360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/089e00e5-7564-469e-8d36-2a8baa86fedc_isrForCrosstalkSources_1125902_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/089e00e5-7564-469e-8d36-2a8baa86fedc_isrForCrosstalkSources_1125902_24.stderr parsl.dataflow.dflow DEBUG: Task 2361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2361 has memoization hash 4b1111b49ca75087403f70537c615f13 parsl.dataflow.memoization INFO: Task 2361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2361 try 0 launched on executor multi with executor id 2362 parsl.dataflow.dflow INFO: Standard output for task 2361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/745bf060-5328-46da-9d64-ea2a85091ce4_isrForCrosstalkSources_1125910_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/745bf060-5328-46da-9d64-ea2a85091ce4_isrForCrosstalkSources_1125910_43.stderr parsl.dataflow.dflow DEBUG: Task 2362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2362 has memoization hash 723661b05f8cf88ba45e3cb43bba5fbf parsl.dataflow.memoization INFO: Task 2362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2362 try 0 launched on executor multi with executor id 2363 parsl.dataflow.dflow INFO: Standard output for task 2362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/a0becab1-a20f-4a7f-804a-6b492771bfb3_isrForCrosstalkSources_1125941_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/a0becab1-a20f-4a7f-804a-6b492771bfb3_isrForCrosstalkSources_1125941_12.stderr parsl.dataflow.dflow DEBUG: Task 2363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2363 has memoization hash 1bd9dc80e1bc325569936acfed84e551 parsl.dataflow.memoization INFO: Task 2363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2363 try 0 launched on executor multi with executor id 2364 parsl.dataflow.dflow INFO: Standard output for task 2363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/5793ea31-f41e-4903-893b-cb088be6aba5_isrForCrosstalkSources_1125910_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/5793ea31-f41e-4903-893b-cb088be6aba5_isrForCrosstalkSources_1125910_62.stderr parsl.dataflow.dflow DEBUG: Task 2364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2364 has memoization hash 19ed3b4611078a4d36b7cc2ebee5f9cb parsl.dataflow.memoization INFO: Task 2364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2364 try 0 launched on executor multi with executor id 2365 parsl.dataflow.dflow INFO: Standard output for task 2364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/287336f5-40ee-42d8-91b4-8a5e1b9edd10_isrForCrosstalkSources_1125915_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/287336f5-40ee-42d8-91b4-8a5e1b9edd10_isrForCrosstalkSources_1125915_24.stderr parsl.dataflow.dflow DEBUG: Task 2365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2365 has memoization hash a6550e3a5ed89e613e8216df4ea9b62f parsl.dataflow.memoization INFO: Task 2365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2365 try 0 launched on executor multi with executor id 2366 parsl.dataflow.dflow INFO: Standard output for task 2365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f2c192b4-2bd9-4de7-b9e9-293c7af8e465_isrForCrosstalkSources_1125945_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f2c192b4-2bd9-4de7-b9e9-293c7af8e465_isrForCrosstalkSources_1125945_28.stderr parsl.dataflow.dflow DEBUG: Task 2366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2366 has memoization hash c55f4b1176916e57ce87d7965f7aabff parsl.dataflow.memoization INFO: Task 2366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2366 try 0 launched on executor multi with executor id 2367 parsl.dataflow.dflow INFO: Standard output for task 2366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/5976ce11-c87f-4e1a-8b0e-70c46075e2f5_isrForCrosstalkSources_1125938_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/5976ce11-c87f-4e1a-8b0e-70c46075e2f5_isrForCrosstalkSources_1125938_55.stderr parsl.dataflow.dflow DEBUG: Task 2367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2367 has memoization hash ac016deb3270351eddd80b36f62b99d6 parsl.dataflow.memoization INFO: Task 2367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2367 try 0 launched on executor multi with executor id 2368 parsl.dataflow.dflow INFO: Standard output for task 2367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/14c30d96-b98b-48bd-a910-bb27c998adf0_isrForCrosstalkSources_1125898_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/14c30d96-b98b-48bd-a910-bb27c998adf0_isrForCrosstalkSources_1125898_39.stderr parsl.dataflow.dflow DEBUG: Task 2368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2368 has memoization hash 3d555cb00c7bb7ac85f24bbd294a7264 parsl.dataflow.memoization INFO: Task 2368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2368 try 0 launched on executor multi with executor id 2369 parsl.dataflow.dflow INFO: Standard output for task 2368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/2a3a5899-9a10-456b-a66c-92e3a451a87c_isrForCrosstalkSources_1125906_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/2a3a5899-9a10-456b-a66c-92e3a451a87c_isrForCrosstalkSources_1125906_17.stderr parsl.dataflow.dflow DEBUG: Task 2369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2369 has memoization hash 435b95fe96073e3a4e4951ddeaf6a25e parsl.dataflow.memoization INFO: Task 2369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2369 try 0 launched on executor multi with executor id 2370 parsl.dataflow.dflow INFO: Standard output for task 2369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/fd3d7c56-1e34-4a20-9d68-2bf8208b229f_isrForCrosstalkSources_1125928_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/fd3d7c56-1e34-4a20-9d68-2bf8208b229f_isrForCrosstalkSources_1125928_12.stderr parsl.dataflow.dflow DEBUG: Task 2370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2370 has memoization hash 4a65423e47c9c751eac4f30f36518179 parsl.dataflow.memoization INFO: Task 2370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2370 try 0 launched on executor multi with executor id 2371 parsl.dataflow.dflow INFO: Standard output for task 2370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/7a273fdb-b05f-4d55-a7f4-2b1bdc0e6924_isrForCrosstalkSources_1125930_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/7a273fdb-b05f-4d55-a7f4-2b1bdc0e6924_isrForCrosstalkSources_1125930_62.stderr parsl.dataflow.dflow DEBUG: Task 2371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2371 has memoization hash 25626d76dac4284fd46df325cc213a9b parsl.dataflow.memoization INFO: Task 2371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2371 try 0 launched on executor multi with executor id 2372 parsl.dataflow.dflow INFO: Standard output for task 2371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/f1bc8a0c-f011-4334-9efa-1751aa3dfa14_isrForCrosstalkSources_1125913_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/f1bc8a0c-f011-4334-9efa-1751aa3dfa14_isrForCrosstalkSources_1125913_7.stderr parsl.dataflow.dflow DEBUG: Task 2372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2372 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2372 has memoization hash fe7c12eb986418d7fafc063648da7263 parsl.dataflow.memoization INFO: Task 2372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2372 try 0 launched on executor multi with executor id 2373 parsl.dataflow.dflow INFO: Standard output for task 2372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/7db01fab-b379-4243-958d-469eef5d01bb_isrForCrosstalkSources_1125892_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/7db01fab-b379-4243-958d-469eef5d01bb_isrForCrosstalkSources_1125892_2.stderr parsl.dataflow.dflow DEBUG: Task 2373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2373 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2373 has memoization hash 3c7252609c87ae19741e15b88797d042 parsl.dataflow.memoization INFO: Task 2373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2373 try 0 launched on executor multi with executor id 2374 parsl.dataflow.dflow INFO: Standard output for task 2373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/fc99383e-006a-4556-8eb5-196ef8b7e906_isrForCrosstalkSources_1125895_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/fc99383e-006a-4556-8eb5-196ef8b7e906_isrForCrosstalkSources_1125895_52.stderr parsl.dataflow.dflow DEBUG: Task 2374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2374 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2374 has memoization hash ea0d00b149bcce53a36bc54bd33252a0 parsl.dataflow.memoization INFO: Task 2374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2374 try 0 launched on executor multi with executor id 2375 parsl.dataflow.dflow INFO: Standard output for task 2374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/1082b762-d8a6-46ba-aade-3dac2b2cbe21_isrForCrosstalkSources_1125943_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/1082b762-d8a6-46ba-aade-3dac2b2cbe21_isrForCrosstalkSources_1125943_6.stderr parsl.dataflow.dflow DEBUG: Task 2375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2375 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2375 has memoization hash 8dc0d29b96fae4facb0b4b0474fe0505 parsl.dataflow.memoization INFO: Task 2375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2375 try 0 launched on executor multi with executor id 2376 parsl.dataflow.dflow INFO: Standard output for task 2375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/ddce7134-76f9-46b7-9d78-5e3c4b677e07_isrForCrosstalkSources_1125926_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/ddce7134-76f9-46b7-9d78-5e3c4b677e07_isrForCrosstalkSources_1125926_6.stderr parsl.dataflow.dflow DEBUG: Task 2376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2376 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2376 has memoization hash 7a102af86395677f9e459a33005251bb parsl.dataflow.memoization INFO: Task 2376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2376 try 0 launched on executor multi with executor id 2377 parsl.dataflow.dflow INFO: Standard output for task 2376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/9d54c593-f9c4-4fa2-9ff8-f17c6a1f96c8_isrForCrosstalkSources_1125919_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/9d54c593-f9c4-4fa2-9ff8-f17c6a1f96c8_isrForCrosstalkSources_1125919_22.stderr parsl.dataflow.dflow DEBUG: Task 2377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2377 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2377 has memoization hash 95db143a43008c4c963b0f7869f53225 parsl.dataflow.memoization INFO: Task 2377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2377 try 0 launched on executor multi with executor id 2378 parsl.dataflow.dflow INFO: Standard output for task 2377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/96ea3005-ba8e-4b96-ab09-927e6791572d_isrForCrosstalkSources_1125946_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/96ea3005-ba8e-4b96-ab09-927e6791572d_isrForCrosstalkSources_1125946_17.stderr parsl.dataflow.dflow DEBUG: Task 2378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2378 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2378 has memoization hash 429c24792b99ace5f2f4515230f64a03 parsl.dataflow.memoization INFO: Task 2378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2378 try 0 launched on executor multi with executor id 2379 parsl.dataflow.dflow INFO: Standard output for task 2378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/f17add48-ac71-4637-b36f-694020352212_isrForCrosstalkSources_1125906_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/f17add48-ac71-4637-b36f-694020352212_isrForCrosstalkSources_1125906_46.stderr parsl.dataflow.dflow DEBUG: Task 2379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2379 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2379 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2379 has memoization hash e91cbfd060f9badaf71bad14ee6c9df0 parsl.dataflow.memoization INFO: Task 2379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2379 try 0 launched on executor multi with executor id 2380 parsl.dataflow.dflow INFO: Standard output for task 2379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/a3ad8fc0-cd5b-4128-b83d-ed8365f94a9e_isrForCrosstalkSources_1125942_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/a3ad8fc0-cd5b-4128-b83d-ed8365f94a9e_isrForCrosstalkSources_1125942_29.stderr parsl.dataflow.dflow DEBUG: Task 2380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2380 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2380 has memoization hash fcf0bf55d54903eaee48bf24a161c1b3 parsl.dataflow.memoization INFO: Task 2380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2380 try 0 launched on executor multi with executor id 2381 parsl.dataflow.dflow INFO: Standard output for task 2380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/7751bd4c-f060-4759-895b-f50ba239b923_isrForCrosstalkSources_1125940_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/7751bd4c-f060-4759-895b-f50ba239b923_isrForCrosstalkSources_1125940_18.stderr parsl.dataflow.dflow DEBUG: Task 2381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2381 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2381 has memoization hash ef6342231bb5595ec31a9d8b35a4c116 parsl.dataflow.memoization INFO: Task 2381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2381 try 0 launched on executor multi with executor id 2382 parsl.dataflow.dflow INFO: Standard output for task 2381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/b781b13b-64cc-4d4b-b2c9-49ce209d0982_isrForCrosstalkSources_1125928_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/b781b13b-64cc-4d4b-b2c9-49ce209d0982_isrForCrosstalkSources_1125928_13.stderr parsl.dataflow.dflow DEBUG: Task 2382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2382 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2382 has memoization hash 076d6abe23421a32900447b3635fcf97 parsl.dataflow.memoization INFO: Task 2382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2382 try 0 launched on executor multi with executor id 2383 parsl.dataflow.dflow INFO: Standard output for task 2382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/d20c75c0-9795-40be-aa58-bd15e8b32de5_isrForCrosstalkSources_1125910_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/d20c75c0-9795-40be-aa58-bd15e8b32de5_isrForCrosstalkSources_1125910_57.stderr parsl.dataflow.dflow DEBUG: Task 2383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2383 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2383 has memoization hash df948c94156dfc5fc02a2289eccae242 parsl.dataflow.memoization INFO: Task 2383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2383 try 0 launched on executor multi with executor id 2384 parsl.dataflow.dflow INFO: Standard output for task 2383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/49477b2f-9cbf-4a6b-b03d-65d9936ac0a2_isrForCrosstalkSources_1125906_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/49477b2f-9cbf-4a6b-b03d-65d9936ac0a2_isrForCrosstalkSources_1125906_11.stderr parsl.dataflow.dflow DEBUG: Task 2384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2384 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2384 has memoization hash d400f30a3143089a0899f6880528fb02 parsl.dataflow.memoization INFO: Task 2384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2384 try 0 launched on executor multi with executor id 2385 parsl.dataflow.dflow INFO: Standard output for task 2384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/c6af7e6a-2885-404d-95fc-04fd73037174_isrForCrosstalkSources_1125924_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/c6af7e6a-2885-404d-95fc-04fd73037174_isrForCrosstalkSources_1125924_56.stderr parsl.dataflow.dflow DEBUG: Task 2385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2385 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2385 has memoization hash f9b792d5ce914b81869190b83e44e6b4 parsl.dataflow.memoization INFO: Task 2385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2385 try 0 launched on executor multi with executor id 2386 parsl.dataflow.dflow INFO: Standard output for task 2385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/077330c3-0182-492c-96e1-232058ddeab0_isrForCrosstalkSources_1125900_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/077330c3-0182-492c-96e1-232058ddeab0_isrForCrosstalkSources_1125900_44.stderr parsl.dataflow.dflow DEBUG: Task 2386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2386 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2386 has memoization hash 83f860a53a19d8cef59099eb70dfc14d parsl.dataflow.memoization INFO: Task 2386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2386 try 0 launched on executor multi with executor id 2387 parsl.dataflow.dflow INFO: Standard output for task 2386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/d32ae83d-6c16-4299-980a-7dcf67e83f15_isrForCrosstalkSources_1125938_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/d32ae83d-6c16-4299-980a-7dcf67e83f15_isrForCrosstalkSources_1125938_53.stderr parsl.dataflow.dflow DEBUG: Task 2387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2387 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2387 has memoization hash 49048ec28eea52acdf8d6d51f04f59ba parsl.dataflow.memoization INFO: Task 2387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2387 try 0 launched on executor multi with executor id 2388 parsl.dataflow.dflow INFO: Standard output for task 2387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/caa117f1-d37a-42dc-8d56-d46c16451d02_isrForCrosstalkSources_1125916_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/caa117f1-d37a-42dc-8d56-d46c16451d02_isrForCrosstalkSources_1125916_10.stderr parsl.dataflow.dflow DEBUG: Task 2388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2388 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2388 has memoization hash ed6303ca7f26f8e048233dca594d7b13 parsl.dataflow.memoization INFO: Task 2388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2388 try 0 launched on executor multi with executor id 2389 parsl.dataflow.dflow INFO: Standard output for task 2388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/bd0a3126-f39d-4043-ac1a-a8523f1b71d0_isrForCrosstalkSources_1125911_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/bd0a3126-f39d-4043-ac1a-a8523f1b71d0_isrForCrosstalkSources_1125911_37.stderr parsl.dataflow.dflow DEBUG: Task 2389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2389 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2389 has memoization hash 640929ac3133f1b3f610b3695961d6ae parsl.dataflow.memoization INFO: Task 2389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2389 try 0 launched on executor multi with executor id 2390 parsl.dataflow.dflow INFO: Standard output for task 2389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d309e542-3e65-46d3-84d9-9c2a5f97bd4e_isrForCrosstalkSources_1125915_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d309e542-3e65-46d3-84d9-9c2a5f97bd4e_isrForCrosstalkSources_1125915_4.stderr parsl.dataflow.dflow DEBUG: Task 2390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2390 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2390 has memoization hash 34530994476b2272e52c09e96e0c710e parsl.dataflow.memoization INFO: Task 2390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2390 try 0 launched on executor multi with executor id 2391 parsl.dataflow.dflow INFO: Standard output for task 2390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/f7f5afd8-272c-47b4-99c1-4dfe5c0ed63d_isrForCrosstalkSources_1125892_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/f7f5afd8-272c-47b4-99c1-4dfe5c0ed63d_isrForCrosstalkSources_1125892_33.stderr parsl.dataflow.dflow DEBUG: Task 2391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2391 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2391 has memoization hash 89475cbfc230825c6457bf330699cc91 parsl.dataflow.memoization INFO: Task 2391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:57 proc_lsst.multi:170 [INFO] [multi] job_id 16896415 proc_lsst.multi INFO: [multi] job_id 16896415 2024-03-07 15:06:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 15:06:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896415 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896415 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16896415 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-07 15:06:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-07 15:06:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 15:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2391 try 0 launched on executor multi with executor id 2392 parsl.dataflow.dflow INFO: Standard output for task 2391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e0f0b5aa-4cf9-4a8e-9e92-946863c27f72_isrForCrosstalkSources_1125942_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e0f0b5aa-4cf9-4a8e-9e92-946863c27f72_isrForCrosstalkSources_1125942_61.stderr parsl.dataflow.dflow DEBUG: Task 2392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2392 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2392 has memoization hash bcea90cc7011a1992835efd856bad60e parsl.dataflow.memoization INFO: Task 2392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2392 try 0 launched on executor multi with executor id 2393 parsl.dataflow.dflow INFO: Standard output for task 2392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c41bb072-b68e-4924-8a47-26ec51784acc_isrForCrosstalkSources_1125917_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c41bb072-b68e-4924-8a47-26ec51784acc_isrForCrosstalkSources_1125917_7.stderr parsl.dataflow.dflow DEBUG: Task 2393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2393 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2393 has memoization hash c5add9a3cd3c9fe3864e021eb01494fc parsl.dataflow.memoization INFO: Task 2393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2393 try 0 launched on executor multi with executor id 2394 parsl.dataflow.dflow INFO: Standard output for task 2393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/24b3c43b-8bb0-4338-ab2a-e267bf703a51_isrForCrosstalkSources_1125922_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/24b3c43b-8bb0-4338-ab2a-e267bf703a51_isrForCrosstalkSources_1125922_4.stderr parsl.dataflow.dflow DEBUG: Task 2394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2394 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2394 has memoization hash 25ec6961034f273cbe447f6e0d7dbcf8 parsl.dataflow.memoization INFO: Task 2394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39a36b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2394 try 0 launched on executor multi with executor id 2395 parsl.dataflow.dflow INFO: Standard output for task 2394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/621e973f-eb18-4620-bde4-b03601aa6306_isrForCrosstalkSources_1125929_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/621e973f-eb18-4620-bde4-b03601aa6306_isrForCrosstalkSources_1125929_59.stderr parsl.dataflow.dflow DEBUG: Task 2395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2395 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2395 has memoization hash 408760f35434a308b5ba2df20b52b147 parsl.dataflow.memoization INFO: Task 2395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2395 try 0 launched on executor multi with executor id 2396 parsl.dataflow.dflow INFO: Standard output for task 2395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/d213eebb-24e4-4ae4-aca3-772a2aa9bccd_isrForCrosstalkSources_1125889_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/d213eebb-24e4-4ae4-aca3-772a2aa9bccd_isrForCrosstalkSources_1125889_54.stderr parsl.dataflow.dflow DEBUG: Task 2396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2396 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2396 has memoization hash 49be3aba8e13a22b7c0a5a0e23ba1956 parsl.dataflow.memoization INFO: Task 2396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2396 try 0 launched on executor multi with executor id 2397 parsl.dataflow.dflow INFO: Standard output for task 2396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/9b994ebf-7934-4d8e-89fa-2f1f0380678b_isrForCrosstalkSources_1125927_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/9b994ebf-7934-4d8e-89fa-2f1f0380678b_isrForCrosstalkSources_1125927_23.stderr parsl.dataflow.dflow DEBUG: Task 2397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2397 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2397 has memoization hash 612c3157b225dcc16c032da5e9406c4f parsl.dataflow.memoization INFO: Task 2397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2397 try 0 launched on executor multi with executor id 2398 parsl.dataflow.dflow INFO: Standard output for task 2397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/52866641-a0c9-431d-9bf6-02d2f9649b27_isrForCrosstalkSources_1125913_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/52866641-a0c9-431d-9bf6-02d2f9649b27_isrForCrosstalkSources_1125913_9.stderr parsl.dataflow.dflow DEBUG: Task 2398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2398 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2398 has memoization hash 8eeed609a8da6095ac82c82cb46b7cda parsl.dataflow.memoization INFO: Task 2398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2398 try 0 launched on executor multi with executor id 2399 parsl.dataflow.dflow INFO: Standard output for task 2398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/f0dcbfa9-0e8b-4936-a6bf-fb515d2d8a37_isrForCrosstalkSources_1125937_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/f0dcbfa9-0e8b-4936-a6bf-fb515d2d8a37_isrForCrosstalkSources_1125937_50.stderr parsl.dataflow.dflow DEBUG: Task 2399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2399 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2399 has memoization hash f7ea46892178613053133d4d966208ac parsl.dataflow.memoization INFO: Task 2399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2399 try 0 launched on executor multi with executor id 2400 parsl.dataflow.dflow INFO: Standard output for task 2399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/728871dd-ed06-491f-8e30-4afd55744b15_isrForCrosstalkSources_1125920_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/728871dd-ed06-491f-8e30-4afd55744b15_isrForCrosstalkSources_1125920_57.stderr parsl.dataflow.dflow DEBUG: Task 2400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2400 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2400 has memoization hash 353cc1c5162d131166f1753cbedd48c6 parsl.dataflow.memoization INFO: Task 2400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2400 try 0 launched on executor multi with executor id 2401 parsl.dataflow.dflow INFO: Standard output for task 2400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/80f6ff20-60d2-45c6-9796-2d16553465de_isrForCrosstalkSources_1125909_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/80f6ff20-60d2-45c6-9796-2d16553465de_isrForCrosstalkSources_1125909_45.stderr parsl.dataflow.dflow DEBUG: Task 2401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2401 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2401 has memoization hash e10dcd5d98766209f393508cbc67ad68 parsl.dataflow.memoization INFO: Task 2401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2401 try 0 launched on executor multi with executor id 2402 parsl.dataflow.dflow INFO: Standard output for task 2401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/500e4ad3-d1f0-4cbb-97d6-18d4ab81ae43_isrForCrosstalkSources_1125899_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/500e4ad3-d1f0-4cbb-97d6-18d4ab81ae43_isrForCrosstalkSources_1125899_34.stderr parsl.dataflow.dflow DEBUG: Task 2402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2402 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2402 has memoization hash ddaabc7994ca0ba5eaf526ba9b65689d parsl.dataflow.memoization INFO: Task 2402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2402 try 0 launched on executor multi with executor id 2403 parsl.dataflow.dflow INFO: Standard output for task 2402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/21d840d1-6ffd-493a-9ae1-dd97e51c2a59_isrForCrosstalkSources_1125911_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/21d840d1-6ffd-493a-9ae1-dd97e51c2a59_isrForCrosstalkSources_1125911_6.stderr parsl.dataflow.dflow DEBUG: Task 2403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2403 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2403 has memoization hash 464190146213162bfe66b9a7314d392e parsl.dataflow.memoization INFO: Task 2403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2403 try 0 launched on executor multi with executor id 2404 parsl.dataflow.dflow INFO: Standard output for task 2403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/5cdacb36-f94e-4760-bc7d-748a686c591d_isrForCrosstalkSources_1125888_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/5cdacb36-f94e-4760-bc7d-748a686c591d_isrForCrosstalkSources_1125888_2.stderr parsl.dataflow.dflow DEBUG: Task 2404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2404 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2404 has memoization hash 0b0e5f556a2c048ac1256916088391fa parsl.dataflow.memoization INFO: Task 2404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2404 try 0 launched on executor multi with executor id 2405 parsl.dataflow.dflow INFO: Standard output for task 2404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/913b7f76-f746-4891-b442-9c7fc6e9af89_isrForCrosstalkSources_1125930_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/913b7f76-f746-4891-b442-9c7fc6e9af89_isrForCrosstalkSources_1125930_34.stderr parsl.dataflow.dflow DEBUG: Task 2405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2405 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2405 has memoization hash 981f44b7ecf30b406e83e4fbc51ef5e2 parsl.dataflow.memoization INFO: Task 2405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2405 try 0 launched on executor multi with executor id 2406 parsl.dataflow.dflow INFO: Standard output for task 2405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/59b9fc98-965b-45ed-9ec3-229f8914a4b9_isrForCrosstalkSources_1125940_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/59b9fc98-965b-45ed-9ec3-229f8914a4b9_isrForCrosstalkSources_1125940_19.stderr parsl.dataflow.dflow DEBUG: Task 2406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2406 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2406 has memoization hash 6b77a3f07679d7a9be4a4710b19bcb33 parsl.dataflow.memoization INFO: Task 2406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2406 try 0 launched on executor multi with executor id 2407 parsl.dataflow.dflow INFO: Standard output for task 2406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/f672bdae-c155-4713-92c6-a2a3b1fa3b60_isrForCrosstalkSources_1125901_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/f672bdae-c155-4713-92c6-a2a3b1fa3b60_isrForCrosstalkSources_1125901_62.stderr parsl.dataflow.dflow DEBUG: Task 2407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2407 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2407 has memoization hash fe53ac664fec9711fa98289115a67a96 parsl.dataflow.memoization INFO: Task 2407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2407 try 0 launched on executor multi with executor id 2408 parsl.dataflow.dflow INFO: Standard output for task 2407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/a5c42aa5-e6cb-47d7-b5d0-96233b1b1555_isrForCrosstalkSources_1125915_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/a5c42aa5-e6cb-47d7-b5d0-96233b1b1555_isrForCrosstalkSources_1125915_18.stderr parsl.dataflow.dflow DEBUG: Task 2408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2408 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2408 has memoization hash 9980f1b93b9e62fceec571e86198f721 parsl.dataflow.memoization INFO: Task 2408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2408 try 0 launched on executor multi with executor id 2409 parsl.dataflow.dflow INFO: Standard output for task 2408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/ec6ac160-6477-43e5-9908-95f22bf5b4d8_isrForCrosstalkSources_1125928_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/ec6ac160-6477-43e5-9908-95f22bf5b4d8_isrForCrosstalkSources_1125928_58.stderr parsl.dataflow.dflow DEBUG: Task 2409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2409 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2409 has memoization hash 73511e08db5cc72f907031bbcc1a4cfc parsl.dataflow.memoization INFO: Task 2409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2409 try 0 launched on executor multi with executor id 2410 parsl.dataflow.dflow INFO: Standard output for task 2409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/bf9bac0c-57ef-4348-b470-e5d6691f39c5_isrForCrosstalkSources_1125947_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/bf9bac0c-57ef-4348-b470-e5d6691f39c5_isrForCrosstalkSources_1125947_50.stderr parsl.dataflow.dflow DEBUG: Task 2410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2410 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2410 has memoization hash 4cb6927575a7533663ab52940b19b3b0 parsl.dataflow.memoization INFO: Task 2410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2410 try 0 launched on executor multi with executor id 2411 parsl.dataflow.dflow INFO: Standard output for task 2410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/6fbf9765-cc24-459d-b804-10b8f6a7e738_isrForCrosstalkSources_1125924_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/6fbf9765-cc24-459d-b804-10b8f6a7e738_isrForCrosstalkSources_1125924_51.stderr parsl.dataflow.dflow DEBUG: Task 2411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2411 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2411 has memoization hash 25ef6985b952e81f0e64ed1cf45a83a6 parsl.dataflow.memoization INFO: Task 2411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2411 try 0 launched on executor multi with executor id 2412 parsl.dataflow.dflow INFO: Standard output for task 2411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/d10563d7-b901-496d-8af9-51da672708a7_isrForCrosstalkSources_1125945_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/d10563d7-b901-496d-8af9-51da672708a7_isrForCrosstalkSources_1125945_35.stderr parsl.dataflow.dflow DEBUG: Task 2412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2412 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2412 has memoization hash 40ad774e174211f9ca158550ea23a3cb parsl.dataflow.memoization INFO: Task 2412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2412 try 0 launched on executor multi with executor id 2413 parsl.dataflow.dflow INFO: Standard output for task 2412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/50da7ea6-2b59-4f45-b92a-c5d378d19a07_isrForCrosstalkSources_1125889_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/50da7ea6-2b59-4f45-b92a-c5d378d19a07_isrForCrosstalkSources_1125889_44.stderr parsl.dataflow.dflow DEBUG: Task 2413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2413 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2413 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2413 has memoization hash 3e6e0a7cb92196ff21e963e649135c46 parsl.dataflow.memoization INFO: Task 2413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2413 try 0 launched on executor multi with executor id 2414 parsl.dataflow.dflow INFO: Standard output for task 2413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d6b689c5-4748-4065-8e37-23da7cefd0e7_isrForCrosstalkSources_1125890_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d6b689c5-4748-4065-8e37-23da7cefd0e7_isrForCrosstalkSources_1125890_62.stderr parsl.dataflow.dflow DEBUG: Task 2414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2414 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2414 has memoization hash 81f875ed614fe7e125a824bf64274c44 parsl.dataflow.memoization INFO: Task 2414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2414 try 0 launched on executor multi with executor id 2415 parsl.dataflow.dflow INFO: Standard output for task 2414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/c8837508-a55e-4c78-af08-fba903f55054_isrForCrosstalkSources_1125937_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/c8837508-a55e-4c78-af08-fba903f55054_isrForCrosstalkSources_1125937_20.stderr parsl.dataflow.dflow DEBUG: Task 2415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2415 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2415 has memoization hash 4c2d30efd7f64795a292354ad9c509b7 parsl.dataflow.memoization INFO: Task 2415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3668382aa555d99eaa1c5e4e8786d999 with result from task 1 parsl.dataflow.dflow INFO: Parsl task 2415 try 0 launched on executor multi with executor id 2416 parsl.dataflow.dflow INFO: Standard output for task 2415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/65d6c609-ebb2-41a0-81ac-636563a38731_isrForCrosstalkSources_1125928_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/65d6c609-ebb2-41a0-81ac-636563a38731_isrForCrosstalkSources_1125928_53.stderr parsl.dataflow.dflow DEBUG: Task 2416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2416 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2416 has memoization hash e8e6c5036c17ba8957358e3de4293042 parsl.dataflow.memoization INFO: Task 2416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2416 try 0 launched on executor multi with executor id 2417 parsl.dataflow.dflow INFO: Standard output for task 2416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/9850f5ad-23f8-4e6a-b6ce-63c6f7db00d5_isrForCrosstalkSources_1125902_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/9850f5ad-23f8-4e6a-b6ce-63c6f7db00d5_isrForCrosstalkSources_1125902_18.stderr parsl.dataflow.dflow DEBUG: Task 2417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2417 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2417 has memoization hash 21beb415a4d13889fd8178bb09f5d94d parsl.dataflow.memoization INFO: Task 2417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/8e6e6a84-b008-4cf0-922c-4a192f7a585e_isrForCrosstalkSources_1125940_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/8e6e6a84-b008-4cf0-922c-4a192f7a585e_isrForCrosstalkSources_1125940_27.stderr parsl.dataflow.dflow INFO: Parsl task 2417 try 0 launched on executor multi with executor id 2418 parsl.dataflow.dflow INFO: Standard output for task 2417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/b2d5eba5-2672-48bb-b0c7-ebd88afc6e4c_isrForCrosstalkSources_1125914_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/b2d5eba5-2672-48bb-b0c7-ebd88afc6e4c_isrForCrosstalkSources_1125914_25.stderr parsl.dataflow.dflow DEBUG: Task 2418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2418 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2418 has memoization hash 8a1f2eb778627dcd01af80960dba7cb3 parsl.dataflow.memoization INFO: Task 2418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2418 try 0 launched on executor multi with executor id 2419 parsl.dataflow.dflow INFO: Standard output for task 2418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/6a0682d9-8e94-4cc1-9d24-3bff04410e33_isrForCrosstalkSources_1125937_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/6a0682d9-8e94-4cc1-9d24-3bff04410e33_isrForCrosstalkSources_1125937_15.stderr parsl.dataflow.dflow DEBUG: Task 2419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2419 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2419 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2419 has memoization hash b035e82bd5cf3e1a48c8f318e1c83bec parsl.dataflow.memoization INFO: Task 2419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2419 try 0 launched on executor multi with executor id 2420 parsl.dataflow.dflow INFO: Standard output for task 2419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/f479768f-27e7-4c49-b2a4-2b9b187411d2_isrForCrosstalkSources_1125895_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/f479768f-27e7-4c49-b2a4-2b9b187411d2_isrForCrosstalkSources_1125895_44.stderr parsl.dataflow.dflow DEBUG: Task 2420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2420 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2420 has memoization hash 5121a19fb809f074f8b6458ac2447ec8 parsl.dataflow.memoization INFO: Task 2420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2420 try 0 launched on executor multi with executor id 2421 parsl.dataflow.dflow INFO: Standard output for task 2420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/4db94df8-c123-4780-b2ca-3882969f5848_isrForCrosstalkSources_1125922_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/4db94df8-c123-4780-b2ca-3882969f5848_isrForCrosstalkSources_1125922_43.stderr parsl.dataflow.dflow DEBUG: Task 2421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2421 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2421 has memoization hash 9229660356e062ede3ac1ac5ab386442 parsl.dataflow.memoization INFO: Task 2421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2421 try 0 launched on executor multi with executor id 2422 parsl.dataflow.dflow INFO: Standard output for task 2421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/28c405d2-a20c-4601-a73b-de68824c444b_isrForCrosstalkSources_1125904_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/28c405d2-a20c-4601-a73b-de68824c444b_isrForCrosstalkSources_1125904_49.stderr parsl.dataflow.dflow DEBUG: Task 2422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2422 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2422 has memoization hash 4e64d31c095709414aa6b91cf0029df8 parsl.dataflow.memoization INFO: Task 2422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2422 try 0 launched on executor multi with executor id 2423 parsl.dataflow.dflow INFO: Standard output for task 2422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/61801768-da6d-4061-8755-95f30d99f13d_isrForCrosstalkSources_1125905_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/61801768-da6d-4061-8755-95f30d99f13d_isrForCrosstalkSources_1125905_21.stderr parsl.dataflow.dflow DEBUG: Task 2423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2423 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2423 has memoization hash 89b18c47c049a29328dcb138e9c4412e parsl.dataflow.memoization INFO: Task 2423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2423 try 0 launched on executor multi with executor id 2424 parsl.dataflow.dflow INFO: Standard output for task 2423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/905a2c2b-4516-4b9e-9af3-cd658f35974a_isrForCrosstalkSources_1125941_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/905a2c2b-4516-4b9e-9af3-cd658f35974a_isrForCrosstalkSources_1125941_3.stderr parsl.dataflow.dflow DEBUG: Task 2424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2424 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2424 has memoization hash 12785d6aa8b23cd29d3d9b5780d853ff parsl.dataflow.memoization INFO: Task 2424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2424 try 0 launched on executor multi with executor id 2425 parsl.dataflow.dflow INFO: Standard output for task 2424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/55c3ed71-9b70-4a86-98f9-fd62b1a3001c_isrForCrosstalkSources_1125933_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/55c3ed71-9b70-4a86-98f9-fd62b1a3001c_isrForCrosstalkSources_1125933_25.stderr parsl.dataflow.dflow DEBUG: Task 2425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2425 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2425 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2425 has memoization hash 4815f7aeec0a87ff988d1ca829e14f62 parsl.dataflow.memoization INFO: Task 2425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2425 try 0 launched on executor multi with executor id 2426 parsl.dataflow.dflow INFO: Standard output for task 2425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/31c5e55c-0446-48ec-be2f-86e6864aacf7_isrForCrosstalkSources_1125923_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/31c5e55c-0446-48ec-be2f-86e6864aacf7_isrForCrosstalkSources_1125923_40.stderr parsl.dataflow.dflow DEBUG: Task 2426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2426 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2426 has memoization hash 82a6258c1d5059fb7668e3386dbfe571 parsl.dataflow.memoization INFO: Task 2426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2426 try 0 launched on executor multi with executor id 2427 parsl.dataflow.dflow INFO: Standard output for task 2426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d043eb23-5dd3-4348-8420-85cb7070b9f4_isrForCrosstalkSources_1125919_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d043eb23-5dd3-4348-8420-85cb7070b9f4_isrForCrosstalkSources_1125919_20.stderr parsl.dataflow.dflow DEBUG: Task 2427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2427 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2427 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2427 has memoization hash 3143c03f291f3c35d5bdbd568e2dc7c2 parsl.dataflow.memoization INFO: Task 2427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2427 try 0 launched on executor multi with executor id 2428 parsl.dataflow.dflow INFO: Standard output for task 2427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/31b23d9a-77c8-425e-b29c-fbc3dd77a16a_isrForCrosstalkSources_1125908_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/31b23d9a-77c8-425e-b29c-fbc3dd77a16a_isrForCrosstalkSources_1125908_33.stderr parsl.dataflow.dflow DEBUG: Task 2428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2428 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2428 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2428 has memoization hash 056703a3efbe0abf45227b8a548d4509 parsl.dataflow.memoization INFO: Task 2428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2428 try 0 launched on executor multi with executor id 2429 parsl.dataflow.dflow INFO: Standard output for task 2428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/61e35de7-0f86-4898-9b2c-bc86ccd7cf93_isrForCrosstalkSources_1125907_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/61e35de7-0f86-4898-9b2c-bc86ccd7cf93_isrForCrosstalkSources_1125907_32.stderr parsl.dataflow.dflow DEBUG: Task 2429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2429 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2429 has memoization hash e3cc4210e4c0b4db51a5e848e5d9ef70 parsl.dataflow.memoization INFO: Task 2429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2429 try 0 launched on executor multi with executor id 2430 parsl.dataflow.dflow INFO: Standard output for task 2429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c2c50ebe-d0cd-4476-b675-bbd918bb908c_isrForCrosstalkSources_1125904_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c2c50ebe-d0cd-4476-b675-bbd918bb908c_isrForCrosstalkSources_1125904_13.stderr parsl.dataflow.dflow DEBUG: Task 2430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2430 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2430 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2430 has memoization hash 8a5d39bdf62719b0a861b14fe63f2f35 parsl.dataflow.memoization INFO: Task 2430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:57 proc_lsst.multi:170 [INFO] [multi] job_id 16896416 proc_lsst.multi INFO: [multi] job_id 16896416 2024-03-07 15:06:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 15:06:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896416 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896416 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16896416 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-07 15:06:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-07 15:06:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 15:06:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2430 try 0 launched on executor multi with executor id 2431 parsl.dataflow.dflow INFO: Standard output for task 2430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/644d53a7-2543-4613-9af6-ecd07f2d43bf_isrForCrosstalkSources_1125911_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/644d53a7-2543-4613-9af6-ecd07f2d43bf_isrForCrosstalkSources_1125911_29.stderr parsl.dataflow.dflow DEBUG: Task 2431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2431 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2431 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2431 has memoization hash de302df3dab6fb2348d775e87bb97de4 parsl.dataflow.memoization INFO: Task 2431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2431 try 0 launched on executor multi with executor id 2432 parsl.dataflow.dflow INFO: Standard output for task 2431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/7c4712a3-29a1-4afa-8704-c19ca8583c83_isrForCrosstalkSources_1125898_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/7c4712a3-29a1-4afa-8704-c19ca8583c83_isrForCrosstalkSources_1125898_9.stderr parsl.dataflow.dflow DEBUG: Task 2432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2432 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2432 has memoization hash dd6cdf86422b2f9d1561f231309b32a9 parsl.dataflow.memoization INFO: Task 2432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2432 try 0 launched on executor multi with executor id 2433 parsl.dataflow.dflow INFO: Standard output for task 2432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/2179fed1-e274-44bd-959a-2d1dd2208e82_isrForCrosstalkSources_1125931_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/2179fed1-e274-44bd-959a-2d1dd2208e82_isrForCrosstalkSources_1125931_37.stderr parsl.dataflow.dflow DEBUG: Task 2433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2433 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2433 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2433 has memoization hash 50a7fbd1a5c5bee3bceadf798aa7bb0d parsl.dataflow.memoization INFO: Task 2433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2433 try 0 launched on executor multi with executor id 2434 parsl.dataflow.dflow INFO: Standard output for task 2433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/30dcf23e-c1e8-44e4-9eba-428dfb098d49_isrForCrosstalkSources_1125889_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/30dcf23e-c1e8-44e4-9eba-428dfb098d49_isrForCrosstalkSources_1125889_3.stderr parsl.dataflow.dflow DEBUG: Task 2434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2434 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2434 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2434 has memoization hash 10fc4ea1d6ff5677dff1be7e229d5c19 parsl.dataflow.memoization INFO: Task 2434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2434 try 0 launched on executor multi with executor id 2435 parsl.dataflow.dflow INFO: Standard output for task 2434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/13dd817d-4a2e-4392-8600-44d71e5b37e6_isrForCrosstalkSources_1125941_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/13dd817d-4a2e-4392-8600-44d71e5b37e6_isrForCrosstalkSources_1125941_51.stderr parsl.dataflow.dflow DEBUG: Task 2435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2435 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2435 has memoization hash beb096b3e947f08a27493c2b4fce1b61 parsl.dataflow.memoization INFO: Task 2435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2435 try 0 launched on executor multi with executor id 2436 parsl.dataflow.dflow INFO: Standard output for task 2435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/b1efc0cf-cc24-4536-a48a-e2975cb6eaa4_isrForCrosstalkSources_1125901_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/b1efc0cf-cc24-4536-a48a-e2975cb6eaa4_isrForCrosstalkSources_1125901_10.stderr parsl.dataflow.dflow DEBUG: Task 2436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2436 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2436 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2436 has memoization hash b83199af4a11beeca2771f2cc4aa849b parsl.dataflow.memoization INFO: Task 2436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2436 try 0 launched on executor multi with executor id 2437 parsl.dataflow.dflow INFO: Standard output for task 2436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/2a39fa7d-a4a1-4bea-a510-b261c7dd9526_isrForCrosstalkSources_1125929_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/2a39fa7d-a4a1-4bea-a510-b261c7dd9526_isrForCrosstalkSources_1125929_18.stderr parsl.dataflow.dflow DEBUG: Task 2437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2437 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2437 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2437 has memoization hash 305109f2d53672d683b99cbc6582acb6 parsl.dataflow.memoization INFO: Task 2437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2437 try 0 launched on executor multi with executor id 2438 parsl.dataflow.dflow INFO: Standard output for task 2437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/13a3eb5a-f8bc-407e-9ae6-6ecb1a8eb08c_isrForCrosstalkSources_1125904_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/13a3eb5a-f8bc-407e-9ae6-6ecb1a8eb08c_isrForCrosstalkSources_1125904_2.stderr parsl.dataflow.dflow DEBUG: Task 2438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2438 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2438 has memoization hash d949eb109951d26b3a4cecdac629ceb3 parsl.dataflow.memoization INFO: Task 2438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2438 try 0 launched on executor multi with executor id 2439 parsl.dataflow.dflow INFO: Standard output for task 2438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/21067d40-d76f-4c02-b340-b2baea0a3470_isrForCrosstalkSources_1125937_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/21067d40-d76f-4c02-b340-b2baea0a3470_isrForCrosstalkSources_1125937_3.stderr parsl.dataflow.dflow DEBUG: Task 2439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2439 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2439 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2439 has memoization hash 69487fcc8153bbb274ed90c90672c1d9 parsl.dataflow.memoization INFO: Task 2439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2439 try 0 launched on executor multi with executor id 2440 parsl.dataflow.dflow INFO: Standard output for task 2439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/c507a99d-1a95-4a21-89d4-3130bdef8350_isrForCrosstalkSources_1125929_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/c507a99d-1a95-4a21-89d4-3130bdef8350_isrForCrosstalkSources_1125929_17.stderr parsl.dataflow.dflow DEBUG: Task 2440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2440 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2440 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2440 has memoization hash ae0277bd258dcff9d8f3329042190bda parsl.dataflow.memoization INFO: Task 2440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2440 try 0 launched on executor multi with executor id 2441 parsl.dataflow.dflow INFO: Standard output for task 2440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/656bab16-5634-4544-bd78-af2bfdbdea4e_isrForCrosstalkSources_1125932_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/656bab16-5634-4544-bd78-af2bfdbdea4e_isrForCrosstalkSources_1125932_51.stderr parsl.dataflow.dflow DEBUG: Task 2441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2441 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2441 has memoization hash 5e7dde622a5c99ba15ad0e87b4eb7464 parsl.dataflow.memoization INFO: Task 2441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2441 try 0 launched on executor multi with executor id 2442 parsl.dataflow.dflow INFO: Standard output for task 2441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/521e0555-f72e-4e8e-9499-146358f7e009_isrForCrosstalkSources_1125913_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/521e0555-f72e-4e8e-9499-146358f7e009_isrForCrosstalkSources_1125913_2.stderr parsl.dataflow.dflow DEBUG: Task 2442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2442 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2442 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2442 has memoization hash 5dcea88e402d566eed834ed591a620d7 parsl.dataflow.memoization INFO: Task 2442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2442 try 0 launched on executor multi with executor id 2443 parsl.dataflow.dflow INFO: Standard output for task 2442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8dcbb2e5-b16d-4fdd-8420-3d952cb93589_isrForCrosstalkSources_1125944_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8dcbb2e5-b16d-4fdd-8420-3d952cb93589_isrForCrosstalkSources_1125944_20.stderr parsl.dataflow.dflow DEBUG: Task 2443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2443 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2443 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2443 has memoization hash 50268e68bbc3f54d39d002651c38c98e parsl.dataflow.memoization INFO: Task 2443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2443 try 0 launched on executor multi with executor id 2444 parsl.dataflow.dflow INFO: Standard output for task 2443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/0f9bde07-8c59-4365-9a78-f0a99411396c_isrForCrosstalkSources_1125946_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/0f9bde07-8c59-4365-9a78-f0a99411396c_isrForCrosstalkSources_1125946_21.stderr parsl.dataflow.dflow DEBUG: Task 2444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2444 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2444 has memoization hash 66eeec3a881a138997b7f3b9e444ea5b parsl.dataflow.memoization INFO: Task 2444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2444 try 0 launched on executor multi with executor id 2445 parsl.dataflow.dflow INFO: Standard output for task 2444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/68116c00-5445-4be7-8c37-4befcd909549_isrForCrosstalkSources_1125942_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/68116c00-5445-4be7-8c37-4befcd909549_isrForCrosstalkSources_1125942_56.stderr parsl.dataflow.dflow DEBUG: Task 2445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2445 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2445 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2445 has memoization hash d20e495c36c2c17c883d4b0d6019944c parsl.dataflow.memoization INFO: Task 2445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2445 try 0 launched on executor multi with executor id 2446 parsl.dataflow.dflow INFO: Standard output for task 2445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/ea3fb236-4114-4239-bc57-52820533029d_isrForCrosstalkSources_1125934_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/ea3fb236-4114-4239-bc57-52820533029d_isrForCrosstalkSources_1125934_7.stderr parsl.dataflow.dflow DEBUG: Task 2446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2446 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2446 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2446 has memoization hash 99e8ac8df428c9b99d86e23372bdacc9 parsl.dataflow.memoization INFO: Task 2446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2446 try 0 launched on executor multi with executor id 2447 parsl.dataflow.dflow INFO: Standard output for task 2446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6d994576-c3f5-40f1-a7a3-3c9b9b73ecfd_isrForCrosstalkSources_1125909_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6d994576-c3f5-40f1-a7a3-3c9b9b73ecfd_isrForCrosstalkSources_1125909_7.stderr parsl.dataflow.dflow DEBUG: Task 2447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2447 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2447 has memoization hash d34b729f3107dd4ff5beab7c9bd8c56e parsl.dataflow.memoization INFO: Task 2447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2447 try 0 launched on executor multi with executor id 2448 parsl.dataflow.dflow INFO: Standard output for task 2447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/69b77c70-64fb-43e5-8d3c-8af96ea15561_isrForCrosstalkSources_1125929_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/69b77c70-64fb-43e5-8d3c-8af96ea15561_isrForCrosstalkSources_1125929_10.stderr parsl.dataflow.dflow DEBUG: Task 2448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2448 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2448 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2448 has memoization hash 82fd566ce0418ae9558897eb0d08e4bc parsl.dataflow.memoization INFO: Task 2448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2448 try 0 launched on executor multi with executor id 2449 parsl.dataflow.dflow INFO: Standard output for task 2448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/228a358e-86b0-45b3-9f64-33221b68e704_isrForCrosstalkSources_1125902_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/228a358e-86b0-45b3-9f64-33221b68e704_isrForCrosstalkSources_1125902_3.stderr parsl.dataflow.dflow DEBUG: Task 2449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2449 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2449 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2449 has memoization hash be5e2aad7a8e20d2030120f59e326dab parsl.dataflow.memoization INFO: Task 2449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2449 try 0 launched on executor multi with executor id 2450 parsl.dataflow.dflow INFO: Standard output for task 2449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/d4322d7e-10ee-4760-92d9-865e74a8d977_isrForCrosstalkSources_1125945_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/d4322d7e-10ee-4760-92d9-865e74a8d977_isrForCrosstalkSources_1125945_22.stderr parsl.dataflow.dflow DEBUG: Task 2450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2450 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2450 has memoization hash 3d8ffed1230dac63247aa93c096eb9a7 parsl.dataflow.memoization INFO: Task 2450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2450 try 0 launched on executor multi with executor id 2451 parsl.dataflow.dflow INFO: Standard output for task 2450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/0de08a5d-3153-440e-8b28-9483a14a9439_isrForCrosstalkSources_1125943_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/0de08a5d-3153-440e-8b28-9483a14a9439_isrForCrosstalkSources_1125943_3.stderr parsl.dataflow.dflow DEBUG: Task 2451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2451 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2451 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2451 has memoization hash 4eab3409bbf50816346fca42a521f1e1 parsl.dataflow.memoization INFO: Task 2451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2451 try 0 launched on executor multi with executor id 2452 parsl.dataflow.dflow INFO: Standard output for task 2451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/138a2a5d-8859-42b4-ba2c-ea364ecc7f06_isrForCrosstalkSources_1125922_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/138a2a5d-8859-42b4-ba2c-ea364ecc7f06_isrForCrosstalkSources_1125922_38.stderr parsl.dataflow.dflow DEBUG: Task 2452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2452 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2452 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2452 has memoization hash 8d8f58aadc698032a591ca6b9fca38c8 parsl.dataflow.memoization INFO: Task 2452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2452 try 0 launched on executor multi with executor id 2453 parsl.dataflow.dflow INFO: Standard output for task 2452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/40e6b5be-8149-48ca-b81a-b5eb35164e4c_isrForCrosstalkSources_1125942_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/40e6b5be-8149-48ca-b81a-b5eb35164e4c_isrForCrosstalkSources_1125942_34.stderr parsl.dataflow.dflow DEBUG: Task 2453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2453 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2453 has memoization hash cc1a536cb90eefebd15e995391fcf538 parsl.dataflow.memoization INFO: Task 2453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2453 try 0 launched on executor multi with executor id 2454 parsl.dataflow.dflow INFO: Standard output for task 2453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/064e9b0c-9810-4d7b-bc3e-dc1b039068d5_isrForCrosstalkSources_1125927_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/064e9b0c-9810-4d7b-bc3e-dc1b039068d5_isrForCrosstalkSources_1125927_7.stderr parsl.dataflow.dflow DEBUG: Task 2454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2454 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2454 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2454 has memoization hash 1bc2e717e892a96f0bb05d8366fda5ef parsl.dataflow.memoization INFO: Task 2454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2454 try 0 launched on executor multi with executor id 2455 parsl.dataflow.dflow INFO: Standard output for task 2454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/8dde435c-ce5a-4ad5-8a5e-c2273392bde8_isrForCrosstalkSources_1125932_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/8dde435c-ce5a-4ad5-8a5e-c2273392bde8_isrForCrosstalkSources_1125932_32.stderr parsl.dataflow.dflow DEBUG: Task 2455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2455 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2455 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2455 has memoization hash 117a07c7eb894ceecd8d6b9aef6dd554 parsl.dataflow.memoization INFO: Task 2455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2455 try 0 launched on executor multi with executor id 2456 parsl.dataflow.dflow INFO: Standard output for task 2455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6ed79345-6a57-4a6c-887a-b33b5811213a_isrForCrosstalkSources_1125927_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6ed79345-6a57-4a6c-887a-b33b5811213a_isrForCrosstalkSources_1125927_61.stderr parsl.dataflow.dflow DEBUG: Task 2456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2456 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2456 has memoization hash 26591f6e3ee38e76fed9eba540509b26 parsl.dataflow.memoization INFO: Task 2456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396342c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2456 try 0 launched on executor multi with executor id 2457 parsl.dataflow.dflow INFO: Standard output for task 2456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/dff1f8be-3913-435c-b625-979b5259599e_isrForCrosstalkSources_1125939_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/dff1f8be-3913-435c-b625-979b5259599e_isrForCrosstalkSources_1125939_34.stderr parsl.dataflow.dflow DEBUG: Task 2457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2457 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2457 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2457 has memoization hash c168671ad6d43913eaedceb521d854e5 parsl.dataflow.memoization INFO: Task 2457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2457 try 0 launched on executor multi with executor id 2458 parsl.dataflow.dflow INFO: Standard output for task 2457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/cbf3578d-a24e-42cd-942e-a11885e41e1d_isrForCrosstalkSources_1125932_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/cbf3578d-a24e-42cd-942e-a11885e41e1d_isrForCrosstalkSources_1125932_7.stderr parsl.dataflow.dflow DEBUG: Task 2458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2458 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2458 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2458 has memoization hash ef9e9bfb02bb40915be7699883c6c309 parsl.dataflow.memoization INFO: Task 2458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396340e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2458 try 0 launched on executor multi with executor id 2459 parsl.dataflow.dflow INFO: Standard output for task 2458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ed783c3e-1a36-40a7-b445-9a77f1db6f46_isrForCrosstalkSources_1125939_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ed783c3e-1a36-40a7-b445-9a77f1db6f46_isrForCrosstalkSources_1125939_15.stderr parsl.dataflow.dflow DEBUG: Task 2459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2459 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2459 has memoization hash 226b38927570eed28b9a18b46aa8eda8 parsl.dataflow.memoization INFO: Task 2459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2459 try 0 launched on executor multi with executor id 2460 parsl.dataflow.dflow INFO: Standard output for task 2459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/f7816967-728e-4127-85ed-e1cfa9278b48_isrForCrosstalkSources_1125891_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/f7816967-728e-4127-85ed-e1cfa9278b48_isrForCrosstalkSources_1125891_36.stderr parsl.dataflow.dflow DEBUG: Task 2460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2460 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2460 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2460 has memoization hash 8d7ca8ede17d62aa08bd075f52c5a598 parsl.dataflow.memoization INFO: Task 2460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2460 try 0 launched on executor multi with executor id 2461 parsl.dataflow.dflow INFO: Standard output for task 2460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f684dfb3-8576-489e-89d3-36026ea34f94_isrForCrosstalkSources_1125894_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f684dfb3-8576-489e-89d3-36026ea34f94_isrForCrosstalkSources_1125894_60.stderr parsl.dataflow.dflow DEBUG: Task 2461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2461 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2461 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2461 has memoization hash 94dda790a1779ca723ff5d61e9e17f5d parsl.dataflow.memoization INFO: Task 2461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2461 try 0 launched on executor multi with executor id 2462 parsl.dataflow.dflow INFO: Standard output for task 2461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4a9f2085-a480-454d-b1a8-618a9883ea4b_isrForCrosstalkSources_1125893_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4a9f2085-a480-454d-b1a8-618a9883ea4b_isrForCrosstalkSources_1125893_10.stderr parsl.dataflow.dflow DEBUG: Task 2462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2462 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2462 has memoization hash 00fe17aefd01f59dd025e7eaf22c2aa7 parsl.dataflow.memoization INFO: Task 2462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396345e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2462 try 0 launched on executor multi with executor id 2463 parsl.dataflow.dflow INFO: Standard output for task 2462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/85c31f09-9c68-4411-83d9-0e03376a941f_isrForCrosstalkSources_1125890_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/85c31f09-9c68-4411-83d9-0e03376a941f_isrForCrosstalkSources_1125890_25.stderr parsl.dataflow.dflow DEBUG: Task 2463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2463 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2463 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2463 has memoization hash 80f6cb488c2330091d7b002eeb0ef63b parsl.dataflow.memoization INFO: Task 2463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2463 try 0 launched on executor multi with executor id 2464 parsl.dataflow.dflow INFO: Standard output for task 2463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/80a5249d-13be-48d3-b500-d8314d3c8e0f_isrForCrosstalkSources_1125889_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/80a5249d-13be-48d3-b500-d8314d3c8e0f_isrForCrosstalkSources_1125889_29.stderr parsl.dataflow.dflow DEBUG: Task 2464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2464 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2464 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2464 has memoization hash 68abebd49d8b914b4056b1e683a950a6 parsl.dataflow.memoization INFO: Task 2464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2464 try 0 launched on executor multi with executor id 2465 parsl.dataflow.dflow INFO: Standard output for task 2464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/a9f0e556-8a48-4209-b486-949109374457_isrForCrosstalkSources_1125917_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/a9f0e556-8a48-4209-b486-949109374457_isrForCrosstalkSources_1125917_4.stderr parsl.dataflow.dflow DEBUG: Task 2465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2465 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2465 has memoization hash f3f747e608b76d32044a431fb7020c58 parsl.dataflow.memoization INFO: Task 2465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396344a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2465 try 0 launched on executor multi with executor id 2466 parsl.dataflow.dflow INFO: Standard output for task 2465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/72d1cd7c-ecff-4b90-b5a7-6c8de8d5b82e_isrForCrosstalkSources_1125894_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/72d1cd7c-ecff-4b90-b5a7-6c8de8d5b82e_isrForCrosstalkSources_1125894_8.stderr parsl.dataflow.dflow DEBUG: Task 2466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2466 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2466 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2466 has memoization hash ff7afdf188f95c021e317de76948ade1 parsl.dataflow.memoization INFO: Task 2466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396347c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2466 try 0 launched on executor multi with executor id 2467 parsl.dataflow.dflow INFO: Standard output for task 2466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/94cc673a-da17-45a9-b7fc-088bd2a4a75f_isrForCrosstalkSources_1125927_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/94cc673a-da17-45a9-b7fc-088bd2a4a75f_isrForCrosstalkSources_1125927_25.stderr parsl.dataflow.dflow DEBUG: Task 2467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2467 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2467 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2467 has memoization hash 64874f529429b439a3acca933fe2e0b4 parsl.dataflow.memoization INFO: Task 2467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2467 try 0 launched on executor multi with executor id 2468 parsl.dataflow.dflow INFO: Standard output for task 2467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/e5a8e964-8a6f-41c4-b3d6-d0b6c184e673_isrForCrosstalkSources_1125889_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/e5a8e964-8a6f-41c4-b3d6-d0b6c184e673_isrForCrosstalkSources_1125889_25.stderr parsl.dataflow.dflow DEBUG: Task 2468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2468 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2468 has memoization hash dc631a221cbf6efe72863094f0692cef parsl.dataflow.memoization INFO: Task 2468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2468 try 0 launched on executor multi with executor id 2469 parsl.dataflow.dflow INFO: Standard output for task 2468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/fb197848-6368-4099-86ae-4be5c3486fc9_isrForCrosstalkSources_1125935_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/fb197848-6368-4099-86ae-4be5c3486fc9_isrForCrosstalkSources_1125935_47.stderr parsl.dataflow.dflow DEBUG: Task 2469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2469 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2469 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2469 has memoization hash 3f4da9cec205234d1f5dff88c4d15adc parsl.dataflow.memoization INFO: Task 2469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396349a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2469 try 0 launched on executor multi with executor id 2470 parsl.dataflow.dflow INFO: Standard output for task 2469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/521a1fbb-28eb-4b79-b056-49669d8f78ef_isrForCrosstalkSources_1125902_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/521a1fbb-28eb-4b79-b056-49669d8f78ef_isrForCrosstalkSources_1125902_37.stderr parsl.dataflow.dflow DEBUG: Task 2470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2470 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2470 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2470 has memoization hash fce3a5807dce6991aeee07b4bbe446fc parsl.dataflow.memoization INFO: Task 2470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2470 try 0 launched on executor multi with executor id 2471 parsl.dataflow.dflow INFO: Standard output for task 2470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/43b4921a-3bd3-47a2-9dbb-91d54e1a97b0_isrForCrosstalkSources_1125900_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/43b4921a-3bd3-47a2-9dbb-91d54e1a97b0_isrForCrosstalkSources_1125900_17.stderr parsl.dataflow.dflow DEBUG: Task 2471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2471 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2471 has memoization hash f507c14a2a9069092b0c8cf7200a5978 parsl.dataflow.memoization INFO: Task 2471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:58 proc_lsst.multi:170 [INFO] [multi] job_id 16896417 proc_lsst.multi INFO: [multi] job_id 16896417 2024-03-07 15:06:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 15:06:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:58 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896417 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896417 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16896417 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-07 15:06:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-07 15:06:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 15:06:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2471 try 0 launched on executor multi with executor id 2472 parsl.dataflow.dflow INFO: Standard output for task 2471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/f75b4542-6f9d-4bdc-addf-7566ae78b5b3_isrForCrosstalkSources_1125912_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/f75b4542-6f9d-4bdc-addf-7566ae78b5b3_isrForCrosstalkSources_1125912_34.stderr parsl.dataflow.dflow DEBUG: Task 2472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2472 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2472 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2472 has memoization hash 3ad39e1aa6c5962cf1e75c5fe0abc4ae parsl.dataflow.memoization INFO: Task 2472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2472 try 0 launched on executor multi with executor id 2473 parsl.dataflow.dflow INFO: Standard output for task 2472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/4c14bdd5-f22d-4090-8db6-c9eeba7ef03c_isrForCrosstalkSources_1125889_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/4c14bdd5-f22d-4090-8db6-c9eeba7ef03c_isrForCrosstalkSources_1125889_37.stderr parsl.dataflow.dflow DEBUG: Task 2473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2473 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2473 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2473 has memoization hash c00cffbb25e6860ba175253fab3b0855 parsl.dataflow.memoization INFO: Task 2473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2473 try 0 launched on executor multi with executor id 2474 parsl.dataflow.dflow INFO: Standard output for task 2473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/2f41b093-6f78-427e-8105-e7f10838c0d9_isrForCrosstalkSources_1125928_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/2f41b093-6f78-427e-8105-e7f10838c0d9_isrForCrosstalkSources_1125928_43.stderr parsl.dataflow.dflow DEBUG: Task 2474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2474 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2474 has memoization hash 0daa5119198c1cf5c5ef121721b432ca parsl.dataflow.memoization INFO: Task 2474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2474 try 0 launched on executor multi with executor id 2475 parsl.dataflow.dflow INFO: Standard output for task 2474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/78a93111-6b11-43ee-b465-b2b6c2441546_isrForCrosstalkSources_1125905_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/78a93111-6b11-43ee-b465-b2b6c2441546_isrForCrosstalkSources_1125905_52.stderr parsl.dataflow.dflow DEBUG: Task 2475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2475 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2475 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2475 has memoization hash 68357bad0601cf757f8f14729ab02b11 parsl.dataflow.memoization INFO: Task 2475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2475 try 0 launched on executor multi with executor id 2476 parsl.dataflow.dflow INFO: Standard output for task 2475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/cec494d1-b7cd-4cf2-bc6e-984830856d10_isrForCrosstalkSources_1125905_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/cec494d1-b7cd-4cf2-bc6e-984830856d10_isrForCrosstalkSources_1125905_12.stderr parsl.dataflow.dflow DEBUG: Task 2476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2476 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2476 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2476 has memoization hash 906e7110e4a26bac8ac32576eae91c88 parsl.dataflow.memoization INFO: Task 2476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2476 try 0 launched on executor multi with executor id 2477 parsl.dataflow.dflow INFO: Standard output for task 2476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/9bd9fb08-e454-4c69-816a-6b9332fb716a_isrForCrosstalkSources_1125919_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/9bd9fb08-e454-4c69-816a-6b9332fb716a_isrForCrosstalkSources_1125919_43.stderr parsl.dataflow.dflow DEBUG: Task 2477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2477 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2477 has memoization hash 2319304926720424c76c251179c6275b parsl.dataflow.memoization INFO: Task 2477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2477 try 0 launched on executor multi with executor id 2478 parsl.dataflow.dflow INFO: Standard output for task 2477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/ff1b01b2-7290-4974-b337-ce7c07be68fa_isrForCrosstalkSources_1125935_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/ff1b01b2-7290-4974-b337-ce7c07be68fa_isrForCrosstalkSources_1125935_60.stderr parsl.dataflow.dflow DEBUG: Task 2478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2478 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2478 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2478 has memoization hash 303e6c98a77be2aec0daebcd6ea5a694 parsl.dataflow.memoization INFO: Task 2478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2478 try 0 launched on executor multi with executor id 2479 parsl.dataflow.dflow INFO: Standard output for task 2478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/2db39b21-a874-4675-824e-d3b4beb3e12e_isrForCrosstalkSources_1125936_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/2db39b21-a874-4675-824e-d3b4beb3e12e_isrForCrosstalkSources_1125936_13.stderr parsl.dataflow.dflow DEBUG: Task 2479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2479 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2479 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2479 has memoization hash 9668a86837740fd9a1a26b1b38685652 parsl.dataflow.memoization INFO: Task 2479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2479 try 0 launched on executor multi with executor id 2480 parsl.dataflow.dflow INFO: Standard output for task 2479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/87c7c04c-9438-48c0-aced-a265ede3cec0_isrForCrosstalkSources_1125944_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/87c7c04c-9438-48c0-aced-a265ede3cec0_isrForCrosstalkSources_1125944_32.stderr parsl.dataflow.dflow DEBUG: Task 2480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2480 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2480 has memoization hash 816356be3711e951c73b98a3f4d86706 parsl.dataflow.memoization INFO: Task 2480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2480 try 0 launched on executor multi with executor id 2481 parsl.dataflow.dflow INFO: Standard output for task 2480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/3421f4f8-2776-40be-ad85-21e5f3568ee6_isrForCrosstalkSources_1125939_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/3421f4f8-2776-40be-ad85-21e5f3568ee6_isrForCrosstalkSources_1125939_8.stderr parsl.dataflow.dflow DEBUG: Task 2481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2481 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2481 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2481 has memoization hash 63ae9fec63d3735e959cafa70702fa6d parsl.dataflow.memoization INFO: Task 2481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2481 try 0 launched on executor multi with executor id 2482 parsl.dataflow.dflow INFO: Standard output for task 2481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/115b6b41-d050-4fd1-934a-891cfbe0d65a_isrForCrosstalkSources_1125922_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/115b6b41-d050-4fd1-934a-891cfbe0d65a_isrForCrosstalkSources_1125922_22.stderr parsl.dataflow.dflow DEBUG: Task 2482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2482 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2482 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2482 has memoization hash 8b24b53a9c285e01c6d7b14f73c7c6c0 parsl.dataflow.memoization INFO: Task 2482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2482 try 0 launched on executor multi with executor id 2483 parsl.dataflow.dflow INFO: Standard output for task 2482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/3f2f922b-a7eb-4f00-8a74-5d8e2833fcd7_isrForCrosstalkSources_1125938_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/3f2f922b-a7eb-4f00-8a74-5d8e2833fcd7_isrForCrosstalkSources_1125938_45.stderr parsl.dataflow.dflow DEBUG: Task 2483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2483 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2483 has memoization hash f2897edd6d2849813ce0e3b7145b4470 parsl.dataflow.memoization INFO: Task 2483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2483 try 0 launched on executor multi with executor id 2484 parsl.dataflow.dflow INFO: Standard output for task 2483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/1241e008-385a-48c3-b7a8-dbd6bd4deecb_isrForCrosstalkSources_1125892_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/1241e008-385a-48c3-b7a8-dbd6bd4deecb_isrForCrosstalkSources_1125892_40.stderr parsl.dataflow.dflow DEBUG: Task 2484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2484 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2484 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2484 has memoization hash 50cdca1eacc53d4e84f8f1e3d161142a parsl.dataflow.memoization INFO: Task 2484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2484 try 0 launched on executor multi with executor id 2485 parsl.dataflow.dflow INFO: Standard output for task 2484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/e85faa8e-dfa7-4f5a-beed-a20dbba2b9dd_isrForCrosstalkSources_1125927_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/e85faa8e-dfa7-4f5a-beed-a20dbba2b9dd_isrForCrosstalkSources_1125927_16.stderr parsl.dataflow.dflow DEBUG: Task 2485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2485 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2485 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2485 has memoization hash bbda9f20f51a293d03c29bb16915f2a6 parsl.dataflow.memoization INFO: Task 2485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2485 try 0 launched on executor multi with executor id 2486 parsl.dataflow.dflow INFO: Standard output for task 2485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/76cf491d-a37b-48a5-afb6-74643f728fa4_isrForCrosstalkSources_1125899_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/76cf491d-a37b-48a5-afb6-74643f728fa4_isrForCrosstalkSources_1125899_46.stderr parsl.dataflow.dflow DEBUG: Task 2486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2486 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2486 has memoization hash e2cc94cdbf825fd99065ae8c7e85b754 parsl.dataflow.memoization INFO: Task 2486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2486 try 0 launched on executor multi with executor id 2487 parsl.dataflow.dflow INFO: Standard output for task 2486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d8cd2914-93e0-45e7-acf2-d24a5736a57d_isrForCrosstalkSources_1125890_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d8cd2914-93e0-45e7-acf2-d24a5736a57d_isrForCrosstalkSources_1125890_51.stderr parsl.dataflow.dflow DEBUG: Task 2487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2487 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2487 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2487 has memoization hash 5f0b7c85b8919361de3ae31a5efeeb9e parsl.dataflow.memoization INFO: Task 2487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396351c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2487 try 0 launched on executor multi with executor id 2488 parsl.dataflow.dflow INFO: Standard output for task 2487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/e4bde4d0-6752-4db6-a463-5f9737424703_isrForCrosstalkSources_1125922_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/e4bde4d0-6752-4db6-a463-5f9737424703_isrForCrosstalkSources_1125922_58.stderr parsl.dataflow.dflow DEBUG: Task 2488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2488 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2488 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2488 has memoization hash b611300201ae3f7818d46cbe4d30eb0a parsl.dataflow.memoization INFO: Task 2488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2488 try 0 launched on executor multi with executor id 2489 parsl.dataflow.dflow INFO: Standard output for task 2488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/04527293-7420-44c4-a9d9-5d8842922ecd_isrForCrosstalkSources_1125889_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/04527293-7420-44c4-a9d9-5d8842922ecd_isrForCrosstalkSources_1125889_56.stderr parsl.dataflow.dflow DEBUG: Task 2489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2489 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2489 has memoization hash 1603f80dcf0e472e0ec06b75c884c34d parsl.dataflow.memoization INFO: Task 2489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396353a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2489 try 0 launched on executor multi with executor id 2490 parsl.dataflow.dflow INFO: Standard output for task 2489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/bee4bf02-711a-4a06-83e7-a4bfba29e491_isrForCrosstalkSources_1125929_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/bee4bf02-711a-4a06-83e7-a4bfba29e491_isrForCrosstalkSources_1125929_62.stderr parsl.dataflow.dflow DEBUG: Task 2490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2490 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2490 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2490 has memoization hash 71d4d200a1eb6754b2f7d0c7ef9d5457 parsl.dataflow.memoization INFO: Task 2490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396354e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2490 try 0 launched on executor multi with executor id 2491 parsl.dataflow.dflow INFO: Standard output for task 2490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/47fb2d24-b45f-4418-8b3a-da929e74f41e_isrForCrosstalkSources_1125914_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/47fb2d24-b45f-4418-8b3a-da929e74f41e_isrForCrosstalkSources_1125914_51.stderr parsl.dataflow.dflow DEBUG: Task 2491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2491 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2491 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2491 has memoization hash 1732c9a91de7678140cfd02f2699b43f parsl.dataflow.memoization INFO: Task 2491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2491 try 0 launched on executor multi with executor id 2492 parsl.dataflow.dflow INFO: Standard output for task 2491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/950ab69e-a603-412e-bcbf-9c9d669a2b58_isrForCrosstalkSources_1125907_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/950ab69e-a603-412e-bcbf-9c9d669a2b58_isrForCrosstalkSources_1125907_57.stderr parsl.dataflow.dflow DEBUG: Task 2492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2492 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2492 has memoization hash 019e5834722d38489f41d8dcf6d18e78 parsl.dataflow.memoization INFO: Task 2492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2492 try 0 launched on executor multi with executor id 2493 parsl.dataflow.dflow INFO: Standard output for task 2492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/46b3790b-cfbf-449e-a525-86db50bd0ff3_isrForCrosstalkSources_1125894_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/46b3790b-cfbf-449e-a525-86db50bd0ff3_isrForCrosstalkSources_1125894_61.stderr parsl.dataflow.dflow DEBUG: Task 2493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2493 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2493 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2493 has memoization hash 382c96736b3512284c72cc30e2ba82db parsl.dataflow.memoization INFO: Task 2493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2493 try 0 launched on executor multi with executor id 2494 parsl.dataflow.dflow INFO: Standard output for task 2493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/e3b96591-6895-45ca-bada-8adc9e6ee66e_isrForCrosstalkSources_1125897_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/e3b96591-6895-45ca-bada-8adc9e6ee66e_isrForCrosstalkSources_1125897_26.stderr parsl.dataflow.dflow DEBUG: Task 2494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2494 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2494 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2494 has memoization hash f269550dc612086ebff72a01ce65068e parsl.dataflow.memoization INFO: Task 2494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2494 try 0 launched on executor multi with executor id 2495 parsl.dataflow.dflow INFO: Standard output for task 2494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/1774d8b4-a0ce-47c3-8b5e-84643d3fd6dc_isrForCrosstalkSources_1125930_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/1774d8b4-a0ce-47c3-8b5e-84643d3fd6dc_isrForCrosstalkSources_1125930_42.stderr parsl.dataflow.dflow DEBUG: Task 2495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2495 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2495 has memoization hash f41aac1bd1a9e2e79061dbd4f632b8eb parsl.dataflow.memoization INFO: Task 2495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2495 try 0 launched on executor multi with executor id 2496 parsl.dataflow.dflow INFO: Standard output for task 2495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/029ba425-fabc-4cfa-a3bf-289b9f9c0d2c_isrForCrosstalkSources_1125903_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/029ba425-fabc-4cfa-a3bf-289b9f9c0d2c_isrForCrosstalkSources_1125903_57.stderr parsl.dataflow.dflow DEBUG: Task 2496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2496 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2496 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2496 has memoization hash 18f4e926ae405a8ead997ec11529bb5e parsl.dataflow.memoization INFO: Task 2496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2496 try 0 launched on executor multi with executor id 2497 parsl.dataflow.dflow INFO: Standard output for task 2496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/9335968f-b7ae-45c6-863f-414fb21106aa_isrForCrosstalkSources_1125929_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/9335968f-b7ae-45c6-863f-414fb21106aa_isrForCrosstalkSources_1125929_28.stderr parsl.dataflow.dflow DEBUG: Task 2497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2497 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2497 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2497 has memoization hash 7ada9eb2c40ecf38b75a9702b520bcfe parsl.dataflow.memoization INFO: Task 2497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396367a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2497 try 0 launched on executor multi with executor id 2498 parsl.dataflow.dflow INFO: Standard output for task 2497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/2b91d6f6-fa57-4371-92e8-83186d90b96c_isrForCrosstalkSources_1125932_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/2b91d6f6-fa57-4371-92e8-83186d90b96c_isrForCrosstalkSources_1125932_3.stderr parsl.dataflow.dflow DEBUG: Task 2498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2498 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2498 has memoization hash a2c714762a10db2ef1f2fe83b868affc parsl.dataflow.memoization INFO: Task 2498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396368e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2498 try 0 launched on executor multi with executor id 2499 parsl.dataflow.dflow INFO: Standard output for task 2498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/6e7bc300-70af-4e4a-aee9-b60fe9cdc294_isrForCrosstalkSources_1125911_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/6e7bc300-70af-4e4a-aee9-b60fe9cdc294_isrForCrosstalkSources_1125911_49.stderr parsl.dataflow.dflow DEBUG: Task 2499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2499 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2499 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2499 has memoization hash 8469caa7c58307b707e268de671ddb54 parsl.dataflow.memoization INFO: Task 2499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2499 try 0 launched on executor multi with executor id 2500 parsl.dataflow.dflow INFO: Standard output for task 2499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/bf0c9cda-b1ea-400e-8d0d-feea68729462_isrForCrosstalkSources_1125932_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/bf0c9cda-b1ea-400e-8d0d-feea68729462_isrForCrosstalkSources_1125932_49.stderr parsl.dataflow.dflow DEBUG: Task 2500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2500 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2500 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2500 has memoization hash 2cb07de37592e25e618d5fa3b6bbda24 parsl.dataflow.memoization INFO: Task 2500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396365c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2500 try 0 launched on executor multi with executor id 2501 parsl.dataflow.dflow INFO: Standard output for task 2500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/d361014e-8336-4fef-8e86-d484eee2355a_isrForCrosstalkSources_1125930_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/d361014e-8336-4fef-8e86-d484eee2355a_isrForCrosstalkSources_1125930_38.stderr parsl.dataflow.dflow DEBUG: Task 2501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2501 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2501 has memoization hash bf01aac9bab26c676fc8e12f6d29c26d parsl.dataflow.memoization INFO: Task 2501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2501 try 0 launched on executor multi with executor id 2502 parsl.dataflow.dflow INFO: Standard output for task 2501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/29c9d42e-39eb-48a5-b94a-d0fa2688cfd4_isrForCrosstalkSources_1125913_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/29c9d42e-39eb-48a5-b94a-d0fa2688cfd4_isrForCrosstalkSources_1125913_37.stderr parsl.dataflow.dflow DEBUG: Task 2502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2502 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2502 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2502 has memoization hash fa36f7db834819a3b396c7e660ad2a85 parsl.dataflow.memoization INFO: Task 2502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2502 try 0 launched on executor multi with executor id 2503 parsl.dataflow.dflow INFO: Standard output for task 2502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/d819c805-b474-4e56-8eb8-2c0c78e84223_isrForCrosstalkSources_1125935_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/d819c805-b474-4e56-8eb8-2c0c78e84223_isrForCrosstalkSources_1125935_50.stderr parsl.dataflow.dflow DEBUG: Task 2503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2503 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2503 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2503 has memoization hash df0fb6214d7bd519bb1f8e124ae5d117 parsl.dataflow.memoization INFO: Task 2503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:58 proc_lsst.multi:170 [INFO] [multi] job_id 16896418 proc_lsst.multi INFO: [multi] job_id 16896418 2024-03-07 15:06:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:06:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:58 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16896418 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16896418 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16896418 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-07 15:06:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-07 15:06:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:06:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 15:06:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:06:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2503 try 0 launched on executor multi with executor id 2504 parsl.dataflow.dflow INFO: Standard output for task 2503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/147b0419-a7db-4bb4-b56d-a6f85210b764_isrForCrosstalkSources_1125945_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/147b0419-a7db-4bb4-b56d-a6f85210b764_isrForCrosstalkSources_1125945_36.stderr parsl.dataflow.dflow DEBUG: Task 2504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2504 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2504 has memoization hash 12048b8dae777dc04bd1607105a9a8c0 parsl.dataflow.memoization INFO: Task 2504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2504 try 0 launched on executor multi with executor id 2505 parsl.dataflow.dflow INFO: Standard output for task 2504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0de1c688-a147-423c-a4d3-bc32fe380326_isrForCrosstalkSources_1125904_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0de1c688-a147-423c-a4d3-bc32fe380326_isrForCrosstalkSources_1125904_33.stderr parsl.dataflow.dflow DEBUG: Task 2505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2505 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2505 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2505 has memoization hash c1765a04a27cc6cbfa669dc32dbcda53 parsl.dataflow.memoization INFO: Task 2505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396363e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2505 try 0 launched on executor multi with executor id 2506 parsl.dataflow.dflow INFO: Standard output for task 2505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/ae2daec1-8f9a-4707-9860-01b230e15cab_isrForCrosstalkSources_1125905_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/ae2daec1-8f9a-4707-9860-01b230e15cab_isrForCrosstalkSources_1125905_18.stderr parsl.dataflow.dflow DEBUG: Task 2506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2506 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2506 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2506 has memoization hash 8fccc38346554a19612100a226f95c15 parsl.dataflow.memoization INFO: Task 2506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2506 try 0 launched on executor multi with executor id 2507 parsl.dataflow.dflow INFO: Standard output for task 2506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/8939f7b3-1620-42bc-95ed-95fce42eff0d_isrForCrosstalkSources_1125900_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/8939f7b3-1620-42bc-95ed-95fce42eff0d_isrForCrosstalkSources_1125900_26.stderr parsl.dataflow.dflow DEBUG: Task 2507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2507 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2507 has memoization hash 31c2b644b5b0e8aaea98decae0dbe279 parsl.dataflow.memoization INFO: Task 2507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2507 try 0 launched on executor multi with executor id 2508 parsl.dataflow.dflow INFO: Standard output for task 2507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/7a95c313-c882-4eae-b6ef-eb09cf6f1f81_isrForCrosstalkSources_1125921_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/7a95c313-c882-4eae-b6ef-eb09cf6f1f81_isrForCrosstalkSources_1125921_38.stderr parsl.dataflow.dflow DEBUG: Task 2508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2508 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2508 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2508 has memoization hash e389ff415e2021926200a47bcb0fcc93 parsl.dataflow.memoization INFO: Task 2508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2508 try 0 launched on executor multi with executor id 2509 parsl.dataflow.dflow INFO: Standard output for task 2508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/e16d4912-8a40-469e-b305-4b0028915fed_isrForCrosstalkSources_1125906_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/e16d4912-8a40-469e-b305-4b0028915fed_isrForCrosstalkSources_1125906_24.stderr parsl.dataflow.dflow DEBUG: Task 2509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2509 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2509 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2509 has memoization hash f1e139976b1c655a50b867262388946b parsl.dataflow.memoization INFO: Task 2509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396362a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2509 try 0 launched on executor multi with executor id 2510 parsl.dataflow.dflow INFO: Standard output for task 2509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ba716670-ba5c-4132-bc34-7ab88f434b57_isrForCrosstalkSources_1125947_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ba716670-ba5c-4132-bc34-7ab88f434b57_isrForCrosstalkSources_1125947_38.stderr parsl.dataflow.dflow DEBUG: Task 2510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2510 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2510 has memoization hash aa15ce2b4902c5192c97e85012ab25fe parsl.dataflow.memoization INFO: Task 2510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396360c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2510 try 0 launched on executor multi with executor id 2511 parsl.dataflow.dflow INFO: Standard output for task 2510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/7412d0d0-c5d8-48d9-9ba7-8f4363baae36_isrForCrosstalkSources_1125889_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/7412d0d0-c5d8-48d9-9ba7-8f4363baae36_isrForCrosstalkSources_1125889_61.stderr parsl.dataflow.dflow DEBUG: Task 2511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2511 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2511 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2511 has memoization hash 4467e5b897911025105c2ec8a0be1240 parsl.dataflow.memoization INFO: Task 2511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2511 try 0 launched on executor multi with executor id 2512 parsl.dataflow.dflow INFO: Standard output for task 2511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/3ec15a3d-5016-4275-ac9c-ff5707232165_isrForCrosstalkSources_1125926_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/3ec15a3d-5016-4275-ac9c-ff5707232165_isrForCrosstalkSources_1125926_61.stderr parsl.dataflow.dflow DEBUG: Task 2512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2512 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2512 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2512 has memoization hash cbf83f0bbe668647cb7f1db179776613 parsl.dataflow.memoization INFO: Task 2512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2512 try 0 launched on executor multi with executor id 2513 parsl.dataflow.dflow INFO: Standard output for task 2512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/dbcc0264-c4ab-4757-823c-528f492ada2f_isrForCrosstalkSources_1125891_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/dbcc0264-c4ab-4757-823c-528f492ada2f_isrForCrosstalkSources_1125891_30.stderr parsl.dataflow.dflow DEBUG: Task 2513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2513 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2513 has memoization hash c39822fb3a155a90d06b78a193a22fe9 parsl.dataflow.memoization INFO: Task 2513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2513 try 0 launched on executor multi with executor id 2514 parsl.dataflow.dflow INFO: Standard output for task 2513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/675e3a70-265d-4612-ba5b-676cd1cec474_isrForCrosstalkSources_1125941_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/675e3a70-265d-4612-ba5b-676cd1cec474_isrForCrosstalkSources_1125941_1.stderr parsl.dataflow.dflow DEBUG: Task 2514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2514 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2514 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2514 has memoization hash 9f4025668ad4b2228496e9f86228f877 parsl.dataflow.memoization INFO: Task 2514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2514 try 0 launched on executor multi with executor id 2515 parsl.dataflow.dflow INFO: Standard output for task 2514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/761ac2b9-8002-48fb-8773-c94aa4794d2c_isrForCrosstalkSources_1125902_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/761ac2b9-8002-48fb-8773-c94aa4794d2c_isrForCrosstalkSources_1125902_11.stderr parsl.dataflow.dflow DEBUG: Task 2515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2515 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2515 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2515 has memoization hash 4b14a6ef933c0dc20019817f50beb415 parsl.dataflow.memoization INFO: Task 2515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2515 try 0 launched on executor multi with executor id 2516 parsl.dataflow.dflow INFO: Standard output for task 2515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/240f66dc-9d21-4449-8198-a7e5950b92d5_isrForCrosstalkSources_1125915_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/240f66dc-9d21-4449-8198-a7e5950b92d5_isrForCrosstalkSources_1125915_13.stderr parsl.dataflow.dflow DEBUG: Task 2516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2516 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2516 has memoization hash f598c57f0ac7b8afd98a92da4b01645c parsl.dataflow.memoization INFO: Task 2516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2516 try 0 launched on executor multi with executor id 2517 parsl.dataflow.dflow INFO: Standard output for task 2516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/54700c93-2b0c-42a1-bf8d-875a58b00008_isrForCrosstalkSources_1125895_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/54700c93-2b0c-42a1-bf8d-875a58b00008_isrForCrosstalkSources_1125895_61.stderr parsl.dataflow.dflow DEBUG: Task 2517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2517 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2517 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2517 has memoization hash 3c6bb005dd6c7643a3000bc9f2a4de09 parsl.dataflow.memoization INFO: Task 2517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2517 try 0 launched on executor multi with executor id 2518 parsl.dataflow.dflow INFO: Standard output for task 2517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/22a20b49-55fe-4788-935c-b9e1bc7b18e1_isrForCrosstalkSources_1125941_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/22a20b49-55fe-4788-935c-b9e1bc7b18e1_isrForCrosstalkSources_1125941_23.stderr parsl.dataflow.dflow DEBUG: Task 2518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2518 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2518 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2518 has memoization hash 7444808bb7f52066ea7fac182cac311c parsl.dataflow.memoization INFO: Task 2518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2518 try 0 launched on executor multi with executor id 2519 parsl.dataflow.dflow INFO: Standard output for task 2518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/0d0f41a8-406e-4931-ad45-cf5ff23eab9f_isrForCrosstalkSources_1125926_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/0d0f41a8-406e-4931-ad45-cf5ff23eab9f_isrForCrosstalkSources_1125926_40.stderr parsl.dataflow.dflow DEBUG: Task 2519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2519 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2519 has memoization hash d04ec33eb1d7b786fcef3d4a90852295 parsl.dataflow.memoization INFO: Task 2519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396359e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2519 try 0 launched on executor multi with executor id 2520 parsl.dataflow.dflow INFO: Standard output for task 2519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/e597966d-4e39-443f-8a21-124bab1cf2ff_isrForCrosstalkSources_1125906_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/e597966d-4e39-443f-8a21-124bab1cf2ff_isrForCrosstalkSources_1125906_34.stderr parsl.dataflow.dflow DEBUG: Task 2520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2520 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2520 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2520 has memoization hash f5687f67a1595b80578a478a3f6bc693 parsl.dataflow.memoization INFO: Task 2520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2520 try 0 launched on executor multi with executor id 2521 parsl.dataflow.dflow INFO: Standard output for task 2520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ad442d0c-b651-4861-a387-f2d7b7d531fc_isrForCrosstalkSources_1125939_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ad442d0c-b651-4861-a387-f2d7b7d531fc_isrForCrosstalkSources_1125939_36.stderr parsl.dataflow.dflow DEBUG: Task 2521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2521 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2521 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2521 has memoization hash eea08e6c58fccd017a67d9b6fbd3c1be parsl.dataflow.memoization INFO: Task 2521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396358a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2521 try 0 launched on executor multi with executor id 2522 parsl.dataflow.dflow INFO: Standard output for task 2521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/65b6429e-5f23-43e9-9256-1625b7acb5d2_isrForCrosstalkSources_1125913_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/65b6429e-5f23-43e9-9256-1625b7acb5d2_isrForCrosstalkSources_1125913_3.stderr parsl.dataflow.dflow DEBUG: Task 2522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2522 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2522 has memoization hash 92ec593f7311d98d9b5d7113f5022563 parsl.dataflow.memoization INFO: Task 2522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396356c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2522 try 0 launched on executor multi with executor id 2523 parsl.dataflow.dflow INFO: Standard output for task 2522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/1ac32902-c64e-4df2-9a6e-a36e2e87c59e_isrForCrosstalkSources_1125933_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/1ac32902-c64e-4df2-9a6e-a36e2e87c59e_isrForCrosstalkSources_1125933_60.stderr parsl.dataflow.dflow DEBUG: Task 2523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2523 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2523 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2523 has memoization hash 0ac866ce06e35782ed389ae53449fe18 parsl.dataflow.memoization INFO: Task 2523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2523 try 0 launched on executor multi with executor id 2524 parsl.dataflow.dflow INFO: Standard output for task 2523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/ab553435-02a2-49ff-a6d1-827030ae8512_isrForCrosstalkSources_1125945_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/ab553435-02a2-49ff-a6d1-827030ae8512_isrForCrosstalkSources_1125945_6.stderr parsl.dataflow.dflow DEBUG: Task 2524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2524 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2524 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2524 has memoization hash 38d6184914f703cf1ee5222aa0b2ef7c parsl.dataflow.memoization INFO: Task 2524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2524 try 0 launched on executor multi with executor id 2525 parsl.dataflow.dflow INFO: Standard output for task 2524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/88d75cd0-1818-4752-82ba-515746641a85_isrForCrosstalkSources_1125925_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/88d75cd0-1818-4752-82ba-515746641a85_isrForCrosstalkSources_1125925_26.stderr parsl.dataflow.dflow DEBUG: Task 2525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2525 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2525 has memoization hash a5607c80bf85fff5f219b144e87e2fe9 parsl.dataflow.memoization INFO: Task 2525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2525 try 0 launched on executor multi with executor id 2526 parsl.dataflow.dflow INFO: Standard output for task 2525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/afb24515-d6cc-4f7e-a44a-42f37240191a_isrForCrosstalkSources_1125933_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/afb24515-d6cc-4f7e-a44a-42f37240191a_isrForCrosstalkSources_1125933_7.stderr parsl.dataflow.dflow DEBUG: Task 2526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2526 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2526 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2526 has memoization hash 83e10268c32a1ee0a67d6c6fc96b7ef6 parsl.dataflow.memoization INFO: Task 2526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2526 try 0 launched on executor multi with executor id 2527 parsl.dataflow.dflow INFO: Standard output for task 2526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/144e6b6a-ca51-4d72-8fcc-7392dccbe9a5_isrForCrosstalkSources_1125929_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/144e6b6a-ca51-4d72-8fcc-7392dccbe9a5_isrForCrosstalkSources_1125929_2.stderr parsl.dataflow.dflow DEBUG: Task 2527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2527 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2527 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2527 has memoization hash cdd1339e66954c942ee330095573bd9f parsl.dataflow.memoization INFO: Task 2527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2527 try 0 launched on executor multi with executor id 2528 parsl.dataflow.dflow INFO: Standard output for task 2527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/f423c108-ab9e-468c-81d7-e2f46737a09f_isrForCrosstalkSources_1125920_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/f423c108-ab9e-468c-81d7-e2f46737a09f_isrForCrosstalkSources_1125920_26.stderr parsl.dataflow.dflow DEBUG: Task 2528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2528 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2528 has memoization hash 675777ced5ddae2e936f169e0d86eaa4 parsl.dataflow.memoization INFO: Task 2528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39634b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2528 try 0 launched on executor multi with executor id 2529 parsl.dataflow.dflow INFO: Standard output for task 2528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/46d135b5-5f14-44bd-b5a4-c8cfceea2f42_isrForCrosstalkSources_1125910_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/46d135b5-5f14-44bd-b5a4-c8cfceea2f42_isrForCrosstalkSources_1125910_7.stderr parsl.dataflow.dflow DEBUG: Task 2529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2529 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2529 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2529 has memoization hash 4e4b1be10c94955dcea96e934d89200b parsl.dataflow.memoization INFO: Task 2529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2529 try 0 launched on executor multi with executor id 2530 parsl.dataflow.dflow INFO: Standard output for task 2529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/0636b521-2675-41e5-a2f0-dc00d1966614_isrForCrosstalkSources_1125910_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/0636b521-2675-41e5-a2f0-dc00d1966614_isrForCrosstalkSources_1125910_8.stderr parsl.dataflow.dflow DEBUG: Task 2530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2530 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2530 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2530 has memoization hash 2bf449f9d1c02c5a1ad589d619283b11 parsl.dataflow.memoization INFO: Task 2530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2530 try 0 launched on executor multi with executor id 2531 parsl.dataflow.dflow INFO: Standard output for task 2530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/8088a4ce-070b-4ab1-aded-146db0f5df96_isrForCrosstalkSources_1125896_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/8088a4ce-070b-4ab1-aded-146db0f5df96_isrForCrosstalkSources_1125896_55.stderr parsl.dataflow.dflow DEBUG: Task 2531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2531 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2531 has memoization hash e0c700c2f2b8fbdb7a4861e1e718c57f parsl.dataflow.memoization INFO: Task 2531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2531 try 0 launched on executor multi with executor id 2532 parsl.dataflow.dflow INFO: Standard output for task 2531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/2a29c9ed-86ec-4e12-ab57-389869ba24a5_isrForCrosstalkSources_1125918_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/2a29c9ed-86ec-4e12-ab57-389869ba24a5_isrForCrosstalkSources_1125918_56.stderr parsl.dataflow.dflow DEBUG: Task 2532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2532 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2532 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2532 has memoization hash 66d2ac731d8ec038adcf13f424a9da28 parsl.dataflow.memoization INFO: Task 2532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2532 try 0 launched on executor multi with executor id 2533 parsl.dataflow.dflow INFO: Standard output for task 2532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/41abb216-fc87-47b4-be0c-f35025a2378f_isrForCrosstalkSources_1125912_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/41abb216-fc87-47b4-be0c-f35025a2378f_isrForCrosstalkSources_1125912_29.stderr parsl.dataflow.dflow DEBUG: Task 2533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2533 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2533 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2533 has memoization hash 080754a28f9723dee49df6e5a129c3b4 parsl.dataflow.memoization INFO: Task 2533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2533 try 0 launched on executor multi with executor id 2534 parsl.dataflow.dflow INFO: Standard output for task 2533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/7ebce183-f6fa-41b0-b92a-ca0d327e87a0_isrForCrosstalkSources_1125893_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/7ebce183-f6fa-41b0-b92a-ca0d327e87a0_isrForCrosstalkSources_1125893_26.stderr parsl.dataflow.dflow DEBUG: Task 2534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2534 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2534 has memoization hash 983e2f6ec0af55f8599cb40e1c3b3fc7 parsl.dataflow.memoization INFO: Task 2534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2534 try 0 launched on executor multi with executor id 2535 parsl.dataflow.dflow INFO: Standard output for task 2534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/6253ff47-8005-4e2a-9c10-845996c683dc_isrForCrosstalkSources_1125906_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/6253ff47-8005-4e2a-9c10-845996c683dc_isrForCrosstalkSources_1125906_22.stderr parsl.dataflow.dflow DEBUG: Task 2535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2535 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2535 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2535 has memoization hash 24c27b74dedec2943ce84cd1ce04299c parsl.dataflow.memoization INFO: Task 2535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396377e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2535 try 0 launched on executor multi with executor id 2536 parsl.dataflow.dflow INFO: Standard output for task 2535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/328c142f-74b0-4df6-a759-79a33283258a_isrForCrosstalkSources_1125908_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/328c142f-74b0-4df6-a759-79a33283258a_isrForCrosstalkSources_1125908_60.stderr parsl.dataflow.dflow DEBUG: Task 2536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2536 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2536 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2536 has memoization hash 96f66a70996e61d2757eed960088f1b1 parsl.dataflow.memoization INFO: Task 2536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396379c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2536 try 0 launched on executor multi with executor id 2537 parsl.dataflow.dflow INFO: Standard output for task 2536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/7cdf782b-52e6-410f-bf4b-91717696acb0_isrForCrosstalkSources_1125897_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/7cdf782b-52e6-410f-bf4b-91717696acb0_isrForCrosstalkSources_1125897_47.stderr parsl.dataflow.dflow DEBUG: Task 2537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2537 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2537 has memoization hash fdd3b2816d8ace0a992aeb4b7ced6b28 parsl.dataflow.memoization INFO: Task 2537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2537 try 0 launched on executor multi with executor id 2538 parsl.dataflow.dflow INFO: Standard output for task 2537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/24d3a7ed-0b0b-436c-8111-658784c0f9e1_isrForCrosstalkSources_1125932_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/24d3a7ed-0b0b-436c-8111-658784c0f9e1_isrForCrosstalkSources_1125932_22.stderr parsl.dataflow.dflow DEBUG: Task 2538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2538 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2538 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2538 has memoization hash 9540f1de5b49ade49665f79f00f5d20c parsl.dataflow.memoization INFO: Task 2538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396376a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2538 try 0 launched on executor multi with executor id 2539 parsl.dataflow.dflow INFO: Standard output for task 2538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/dca9be6a-76be-4be3-bc5a-ae2eea0295fa_isrForCrosstalkSources_1125896_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/dca9be6a-76be-4be3-bc5a-ae2eea0295fa_isrForCrosstalkSources_1125896_50.stderr parsl.dataflow.dflow DEBUG: Task 2539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2539 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2539 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2539 has memoization hash 819d8da36ed81c270bbfb19fa0e0e3c0 parsl.dataflow.memoization INFO: Task 2539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:58 proc_lsst.multi:170 [INFO] [multi] job_id 16896419 proc_lsst.multi INFO: [multi] job_id 16896419 2024-03-07 15:06:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:06:58 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16896419 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16896419 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 16896419 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-07 15:06:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-07 15:06:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:06:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:06:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2539 try 0 launched on executor multi with executor id 2540 parsl.dataflow.dflow INFO: Standard output for task 2539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/46adedc5-9faf-4646-b62e-b4bb42bce309_isrForCrosstalkSources_1125943_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/46adedc5-9faf-4646-b62e-b4bb42bce309_isrForCrosstalkSources_1125943_50.stderr parsl.dataflow.dflow DEBUG: Task 2540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2540 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2540 has memoization hash 897ec9723028779a7f67ffa16b30f1ff parsl.dataflow.memoization INFO: Task 2540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396374c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2540 try 0 launched on executor multi with executor id 2541 parsl.dataflow.dflow INFO: Standard output for task 2540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/fe969b72-4e65-4613-a851-1830b2b3a476_isrForCrosstalkSources_1125932_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/fe969b72-4e65-4613-a851-1830b2b3a476_isrForCrosstalkSources_1125932_23.stderr parsl.dataflow.dflow DEBUG: Task 2541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2541 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2541 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2541 has memoization hash d138aeeffdb50cf5a9b456adb8590b61 parsl.dataflow.memoization INFO: Task 2541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2541 try 0 launched on executor multi with executor id 2542 parsl.dataflow.dflow INFO: Standard output for task 2541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/15076fc2-558e-44ec-bd64-e787fa69ede8_isrForCrosstalkSources_1125906_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/15076fc2-558e-44ec-bd64-e787fa69ede8_isrForCrosstalkSources_1125906_23.stderr parsl.dataflow.dflow DEBUG: Task 2542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2542 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2542 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2542 has memoization hash e2632c1e917e30ca23d138b299695713 parsl.dataflow.memoization INFO: Task 2542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2542 try 0 launched on executor multi with executor id 2543 parsl.dataflow.dflow INFO: Standard output for task 2542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/96171b26-176a-463c-a0e6-eb1e4e690349_isrForCrosstalkSources_1125908_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/96171b26-176a-463c-a0e6-eb1e4e690349_isrForCrosstalkSources_1125908_14.stderr parsl.dataflow.dflow DEBUG: Task 2543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2543 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2543 has memoization hash 25b415081aa6beca52593590693cd47d parsl.dataflow.memoization INFO: Task 2543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2543 try 0 launched on executor multi with executor id 2544 parsl.dataflow.dflow INFO: Standard output for task 2543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/db2638f6-1c93-4415-b8dd-e67f14ccf471_isrForCrosstalkSources_1125923_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/db2638f6-1c93-4415-b8dd-e67f14ccf471_isrForCrosstalkSources_1125923_13.stderr parsl.dataflow.dflow DEBUG: Task 2544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2544 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2544 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2544 has memoization hash 5505984ac1738260402beebe66fba1c4 parsl.dataflow.memoization INFO: Task 2544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2544 try 0 launched on executor multi with executor id 2545 parsl.dataflow.dflow INFO: Standard output for task 2544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/112f07d5-2dae-4007-96d4-fbec7942d516_isrForCrosstalkSources_1125943_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/112f07d5-2dae-4007-96d4-fbec7942d516_isrForCrosstalkSources_1125943_1.stderr parsl.dataflow.dflow DEBUG: Task 2545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2545 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2545 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2545 has memoization hash e0c35d43e5c23edd8aea6d9761f33474 parsl.dataflow.memoization INFO: Task 2545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2545 try 0 launched on executor multi with executor id 2546 parsl.dataflow.dflow INFO: Standard output for task 2545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/f786fdad-9902-40bb-befa-24a57e596f3f_isrForCrosstalkSources_1125920_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/f786fdad-9902-40bb-befa-24a57e596f3f_isrForCrosstalkSources_1125920_45.stderr parsl.dataflow.dflow DEBUG: Task 2546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2546 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2546 has memoization hash ac0b5cc351eaee6adeb5cb2903d1f45e parsl.dataflow.memoization INFO: Task 2546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2546 try 0 launched on executor multi with executor id 2547 parsl.dataflow.dflow INFO: Standard output for task 2546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/984dd845-afff-4383-8e6a-58d8135a991a_isrForCrosstalkSources_1125893_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/984dd845-afff-4383-8e6a-58d8135a991a_isrForCrosstalkSources_1125893_56.stderr parsl.dataflow.dflow DEBUG: Task 2547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2547 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2547 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2547 has memoization hash 2417cdd03bd41b1a598fc1788a7b651f parsl.dataflow.memoization INFO: Task 2547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396371a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2547 try 0 launched on executor multi with executor id 2548 parsl.dataflow.dflow INFO: Standard output for task 2547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/d5d6258d-1c90-4df7-9e9f-85c9f49ae570_isrForCrosstalkSources_1125898_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/d5d6258d-1c90-4df7-9e9f-85c9f49ae570_isrForCrosstalkSources_1125898_5.stderr parsl.dataflow.dflow DEBUG: Task 2548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2548 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2548 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2548 has memoization hash 4d7329278a1ee72563ee55d7259bbae2 parsl.dataflow.memoization INFO: Task 2548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396372e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2548 try 0 launched on executor multi with executor id 2549 parsl.dataflow.dflow INFO: Standard output for task 2548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/191bad66-7490-49c1-8772-8cd48cf56750_isrForCrosstalkSources_1125892_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/191bad66-7490-49c1-8772-8cd48cf56750_isrForCrosstalkSources_1125892_1.stderr parsl.dataflow.dflow DEBUG: Task 2549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2549 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2549 has memoization hash f28cee63fe14f3cbaa62ac058b05bc80 parsl.dataflow.memoization INFO: Task 2549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39635bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2549 try 0 launched on executor multi with executor id 2550 parsl.dataflow.dflow INFO: Standard output for task 2549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/926771e2-b228-40c8-af7a-1a853f5caefc_isrForCrosstalkSources_1125920_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/926771e2-b228-40c8-af7a-1a853f5caefc_isrForCrosstalkSources_1125920_36.stderr parsl.dataflow.dflow DEBUG: Task 2550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2550 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2550 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2550 has memoization hash e243b372a7bdcb3dd9e9200bc1640549 parsl.dataflow.memoization INFO: Task 2550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2550 try 0 launched on executor multi with executor id 2551 parsl.dataflow.dflow INFO: Standard output for task 2550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9ed5a1e6-1ad6-43d6-9a49-08122e1c568b_isrForCrosstalkSources_1125934_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9ed5a1e6-1ad6-43d6-9a49-08122e1c568b_isrForCrosstalkSources_1125934_19.stderr parsl.dataflow.dflow DEBUG: Task 2551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2551 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2551 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2551 has memoization hash c29f6074cafe7b89c88b44824334e057 parsl.dataflow.memoization INFO: Task 2551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2551 try 0 launched on executor multi with executor id 2552 parsl.dataflow.dflow INFO: Standard output for task 2551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/50e31278-e651-4427-a6e2-29f051121416_isrForCrosstalkSources_1125934_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/50e31278-e651-4427-a6e2-29f051121416_isrForCrosstalkSources_1125934_61.stderr parsl.dataflow.dflow DEBUG: Task 2552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2552 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2552 has memoization hash c46f667aa1e9f85e6369f9f568a27450 parsl.dataflow.memoization INFO: Task 2552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2552 try 0 launched on executor multi with executor id 2553 parsl.dataflow.dflow INFO: Standard output for task 2552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/1bddabb1-0ff1-4ac0-9d83-f20742c5df63_isrForCrosstalkSources_1125930_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/1bddabb1-0ff1-4ac0-9d83-f20742c5df63_isrForCrosstalkSources_1125930_33.stderr parsl.dataflow.dflow DEBUG: Task 2553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2553 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2553 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2553 has memoization hash 1ba73aaad8b8f22d914f7503188c3d26 parsl.dataflow.memoization INFO: Task 2553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2553 try 0 launched on executor multi with executor id 2554 parsl.dataflow.dflow INFO: Standard output for task 2553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/b9c24582-d89d-4fad-9557-f19cc28857d5_isrForCrosstalkSources_1125928_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/b9c24582-d89d-4fad-9557-f19cc28857d5_isrForCrosstalkSources_1125928_18.stderr parsl.dataflow.dflow DEBUG: Task 2554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2554 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2554 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2554 has memoization hash 816bcb149a128027679bd1a23a6efbb9 parsl.dataflow.memoization INFO: Task 2554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2554 try 0 launched on executor multi with executor id 2555 parsl.dataflow.dflow INFO: Standard output for task 2554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/efc06221-15ed-45a6-a7f9-ebbec51867ff_isrForCrosstalkSources_1125890_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/efc06221-15ed-45a6-a7f9-ebbec51867ff_isrForCrosstalkSources_1125890_4.stderr parsl.dataflow.dflow DEBUG: Task 2555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2555 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2555 has memoization hash 286aeef9767c3c1e2a26b2afc2e01230 parsl.dataflow.memoization INFO: Task 2555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39636f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2555 try 0 launched on executor multi with executor id 2556 parsl.dataflow.dflow INFO: Standard output for task 2555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/0167ecd7-e74b-4838-a6c3-8108181e53af_isrForCrosstalkSources_1125929_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/0167ecd7-e74b-4838-a6c3-8108181e53af_isrForCrosstalkSources_1125929_27.stderr parsl.dataflow.dflow DEBUG: Task 2556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2556 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2556 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2556 has memoization hash 84c8bd06f74a276767d19589b5222642 parsl.dataflow.memoization INFO: Task 2556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396dafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2556 try 0 launched on executor multi with executor id 2557 parsl.dataflow.dflow INFO: Standard output for task 2556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/3b34953c-8c50-4ac9-9783-0c7d64b8ff1e_isrForCrosstalkSources_1125933_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/3b34953c-8c50-4ac9-9783-0c7d64b8ff1e_isrForCrosstalkSources_1125933_50.stderr parsl.dataflow.dflow DEBUG: Task 2557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2557 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2557 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2557 has memoization hash bf64b0df06fcb7a63e9f612ade6a913f parsl.dataflow.memoization INFO: Task 2557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2557 try 0 launched on executor multi with executor id 2558 parsl.dataflow.dflow INFO: Standard output for task 2557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/8befcf5a-91f2-41fa-a504-6fca966fe58a_isrForCrosstalkSources_1125901_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/8befcf5a-91f2-41fa-a504-6fca966fe58a_isrForCrosstalkSources_1125901_11.stderr parsl.dataflow.dflow DEBUG: Task 2558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2558 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2558 has memoization hash 5c4fb015aa59c416624a7c58a69b5430 parsl.dataflow.memoization INFO: Task 2558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396daf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2558 try 0 launched on executor multi with executor id 2559 parsl.dataflow.dflow INFO: Standard output for task 2558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/9b88cf1c-efe8-41e3-9961-e2ce74624642_isrForCrosstalkSources_1125912_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/9b88cf1c-efe8-41e3-9961-e2ce74624642_isrForCrosstalkSources_1125912_51.stderr parsl.dataflow.dflow DEBUG: Task 2559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2559 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2559 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2559 has memoization hash b4b7cfdf197647d06d3435d2b6e08ca6 parsl.dataflow.memoization INFO: Task 2559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2559 try 0 launched on executor multi with executor id 2560 parsl.dataflow.dflow INFO: Standard output for task 2559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/fb724fb9-9ebd-43f3-8e53-bdcb9e7195aa_isrForCrosstalkSources_1125944_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/fb724fb9-9ebd-43f3-8e53-bdcb9e7195aa_isrForCrosstalkSources_1125944_35.stderr parsl.dataflow.dflow DEBUG: Task 2560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2560 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2560 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2560 has memoization hash 5c631fd7b5c1a1e038e50c7aeff5fe25 parsl.dataflow.memoization INFO: Task 2560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2560 try 0 launched on executor multi with executor id 2561 parsl.dataflow.dflow INFO: Standard output for task 2560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/198e3209-0585-4c6d-bd96-ab9b77983213_isrForCrosstalkSources_1125931_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/198e3209-0585-4c6d-bd96-ab9b77983213_isrForCrosstalkSources_1125931_42.stderr parsl.dataflow.dflow DEBUG: Task 2561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2561 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2561 has memoization hash 4be93959808b04dce9794d497929ec8a parsl.dataflow.memoization INFO: Task 2561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2561 try 0 launched on executor multi with executor id 2562 parsl.dataflow.dflow INFO: Standard output for task 2561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/5b78bc60-fb05-4f8d-a209-9a54726f1905_isrForCrosstalkSources_1125906_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/5b78bc60-fb05-4f8d-a209-9a54726f1905_isrForCrosstalkSources_1125906_6.stderr parsl.dataflow.dflow DEBUG: Task 2562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2562 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2562 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2562 has memoization hash 8640b58d843571300204d90b0f0f8115 parsl.dataflow.memoization INFO: Task 2562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2562 try 0 launched on executor multi with executor id 2563 parsl.dataflow.dflow INFO: Standard output for task 2562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2ebbdc4b-c2a3-4da9-9d41-29d6398c682d_isrForCrosstalkSources_1125898_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2ebbdc4b-c2a3-4da9-9d41-29d6398c682d_isrForCrosstalkSources_1125898_52.stderr parsl.dataflow.dflow DEBUG: Task 2563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2563 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2563 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2563 has memoization hash 4e91e7121e90728b8ec5c254a611c442 parsl.dataflow.memoization INFO: Task 2563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2563 try 0 launched on executor multi with executor id 2564 parsl.dataflow.dflow INFO: Standard output for task 2563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/6f35a36c-32ee-4568-8a0c-36fe9cb4ea2a_isrForCrosstalkSources_1125945_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/6f35a36c-32ee-4568-8a0c-36fe9cb4ea2a_isrForCrosstalkSources_1125945_33.stderr parsl.dataflow.dflow DEBUG: Task 2564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2564 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2564 has memoization hash 45fcd95362553490b2ab37b8e945e353 parsl.dataflow.memoization INFO: Task 2564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2564 try 0 launched on executor multi with executor id 2565 parsl.dataflow.dflow INFO: Standard output for task 2564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/13c38b60-e34d-4a9e-b659-a6a058844093_isrForCrosstalkSources_1125944_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/13c38b60-e34d-4a9e-b659-a6a058844093_isrForCrosstalkSources_1125944_55.stderr parsl.dataflow.dflow DEBUG: Task 2565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2565 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2565 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2565 has memoization hash 543aab3c9d7b35741bea576853d9fdb1 parsl.dataflow.memoization INFO: Task 2565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2565 try 0 launched on executor multi with executor id 2566 parsl.dataflow.dflow INFO: Standard output for task 2565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/8827f50f-d79b-41ea-8956-29287a67a619_isrForCrosstalkSources_1125943_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/8827f50f-d79b-41ea-8956-29287a67a619_isrForCrosstalkSources_1125943_42.stderr parsl.dataflow.dflow DEBUG: Task 2566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2566 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2566 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2566 has memoization hash 0c6cf6b1e746f1bd28525aaa3dfbdb75 parsl.dataflow.memoization INFO: Task 2566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2566 try 0 launched on executor multi with executor id 2567 parsl.dataflow.dflow INFO: Standard output for task 2566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/24018173-820a-4691-bea7-747542a20100_isrForCrosstalkSources_1125918_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/24018173-820a-4691-bea7-747542a20100_isrForCrosstalkSources_1125918_3.stderr parsl.dataflow.dflow DEBUG: Task 2567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2567 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2567 has memoization hash c373ceb8160814bd9e3d1b18a8743bf1 parsl.dataflow.memoization INFO: Task 2567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2567 try 0 launched on executor multi with executor id 2568 parsl.dataflow.dflow INFO: Standard output for task 2567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/bd0d4298-4fca-4232-b08b-ada6ca1fc3d2_isrForCrosstalkSources_1125916_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/bd0d4298-4fca-4232-b08b-ada6ca1fc3d2_isrForCrosstalkSources_1125916_38.stderr parsl.dataflow.dflow DEBUG: Task 2568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2568 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2568 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2568 has memoization hash cdfe580d23fb818c9f065c2e6db9bf8d parsl.dataflow.memoization INFO: Task 2568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2568 try 0 launched on executor multi with executor id 2569 parsl.dataflow.dflow INFO: Standard output for task 2568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/633f78dd-9103-43fc-bf35-07895d35f59a_isrForCrosstalkSources_1125946_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/633f78dd-9103-43fc-bf35-07895d35f59a_isrForCrosstalkSources_1125946_44.stderr parsl.dataflow.dflow DEBUG: Task 2569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2569 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2569 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2569 has memoization hash 50cea363ca1ebd3ad49eb36560deb865 parsl.dataflow.memoization INFO: Task 2569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2569 try 0 launched on executor multi with executor id 2570 parsl.dataflow.dflow INFO: Standard output for task 2569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/2689b0f5-286d-4333-981e-21b673badbf7_isrForCrosstalkSources_1125935_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/2689b0f5-286d-4333-981e-21b673badbf7_isrForCrosstalkSources_1125935_45.stderr parsl.dataflow.dflow DEBUG: Task 2570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2570 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2570 has memoization hash 53d4d2a84d9ef64ae6331e612fc65509 parsl.dataflow.memoization INFO: Task 2570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2570 try 0 launched on executor multi with executor id 2571 parsl.dataflow.dflow INFO: Standard output for task 2570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/1043d64a-f2f6-42cd-9751-7b7e5ceeff2d_isrForCrosstalkSources_1125938_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/1043d64a-f2f6-42cd-9751-7b7e5ceeff2d_isrForCrosstalkSources_1125938_61.stderr parsl.dataflow.dflow DEBUG: Task 2571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2571 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2571 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2571 has memoization hash 9a3677275765676507390eba7adebac4 parsl.dataflow.memoization INFO: Task 2571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2571 try 0 launched on executor multi with executor id 2572 parsl.dataflow.dflow INFO: Standard output for task 2571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/f84beaf5-dcb0-48b4-9657-de426099afbe_isrForCrosstalkSources_1125940_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/f84beaf5-dcb0-48b4-9657-de426099afbe_isrForCrosstalkSources_1125940_62.stderr parsl.dataflow.dflow DEBUG: Task 2572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2572 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2572 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2572 has memoization hash 3a89ceb3ce851a3af6544c797aeba03e parsl.dataflow.memoization INFO: Task 2572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2572 try 0 launched on executor multi with executor id 2573 parsl.dataflow.dflow INFO: Standard output for task 2572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/d8335c3b-7c76-4974-aaa3-7a1ee05569c6_isrForCrosstalkSources_1125930_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/d8335c3b-7c76-4974-aaa3-7a1ee05569c6_isrForCrosstalkSources_1125930_10.stderr parsl.dataflow.dflow DEBUG: Task 2573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2573 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2573 has memoization hash 047a64a71ce0dd7889d0d67b40f96f34 parsl.dataflow.memoization INFO: Task 2573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2573 try 0 launched on executor multi with executor id 2574 parsl.dataflow.dflow INFO: Standard output for task 2573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/85c10834-ff9c-4740-a0aa-51e62eeedb7d_isrForCrosstalkSources_1125896_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/85c10834-ff9c-4740-a0aa-51e62eeedb7d_isrForCrosstalkSources_1125896_15.stderr parsl.dataflow.dflow DEBUG: Task 2574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2574 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2574 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2574 has memoization hash 8b3d67d7625a8686154bca08ca083093 parsl.dataflow.memoization INFO: Task 2574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2574 try 0 launched on executor multi with executor id 2575 parsl.dataflow.dflow INFO: Standard output for task 2574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/48bb9258-ec16-4fb6-a804-8f070fba57dd_isrForCrosstalkSources_1125942_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/48bb9258-ec16-4fb6-a804-8f070fba57dd_isrForCrosstalkSources_1125942_23.stderr parsl.dataflow.dflow DEBUG: Task 2575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2575 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2575 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2575 has memoization hash e5dfe8c3b88bb40fcbe601dc1038591e parsl.dataflow.memoization INFO: Task 2575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2575 try 0 launched on executor multi with executor id 2576 parsl.dataflow.dflow INFO: Standard output for task 2575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/e1f58907-8a96-4aa2-8f71-f75092c94059_isrForCrosstalkSources_1125903_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/e1f58907-8a96-4aa2-8f71-f75092c94059_isrForCrosstalkSources_1125903_28.stderr parsl.dataflow.dflow DEBUG: Task 2576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2576 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2576 has memoization hash 31707c9c8ed829284fd144639a5e92f2 parsl.dataflow.memoization INFO: Task 2576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2576 try 0 launched on executor multi with executor id 2577 parsl.dataflow.dflow INFO: Standard output for task 2576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/f44adea6-ae9b-4181-a970-b54ea8192961_isrForCrosstalkSources_1125907_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/f44adea6-ae9b-4181-a970-b54ea8192961_isrForCrosstalkSources_1125907_40.stderr parsl.dataflow.dflow DEBUG: Task 2577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2577 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2577 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2577 has memoization hash 4a2c108ac9b01d5a11ce8bcd91819f9a parsl.dataflow.memoization INFO: Task 2577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2577 try 0 launched on executor multi with executor id 2578 parsl.dataflow.dflow INFO: Standard output for task 2577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/6081de18-f174-4e94-9584-2ffffd77e0ef_isrForCrosstalkSources_1125935_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/6081de18-f174-4e94-9584-2ffffd77e0ef_isrForCrosstalkSources_1125935_21.stderr parsl.dataflow.dflow DEBUG: Task 2578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2578 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2578 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2578 has memoization hash 1e3498cab5241a77ec78dec41361c6e8 parsl.dataflow.memoization INFO: Task 2578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2578 try 0 launched on executor multi with executor id 2579 parsl.dataflow.dflow INFO: Standard output for task 2578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/b28378ae-e658-4748-aebd-230006eabb3c_isrForCrosstalkSources_1125905_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/b28378ae-e658-4748-aebd-230006eabb3c_isrForCrosstalkSources_1125905_1.stderr parsl.dataflow.dflow DEBUG: Task 2579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2579 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2579 has memoization hash a8a3aef00673944bc797d37022fb5880 parsl.dataflow.memoization INFO: Task 2579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2579 try 0 launched on executor multi with executor id 2580 parsl.dataflow.dflow INFO: Standard output for task 2579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/03e4f45c-7794-41a0-a601-85536602a8aa_isrForCrosstalkSources_1125943_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/03e4f45c-7794-41a0-a601-85536602a8aa_isrForCrosstalkSources_1125943_35.stderr parsl.dataflow.dflow DEBUG: Task 2580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2580 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2580 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2580 has memoization hash bd47436391be544990792892fd42c19a parsl.dataflow.memoization INFO: Task 2580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:59 proc_lsst.multi:170 [INFO] [multi] job_id 16896420 proc_lsst.multi INFO: [multi] job_id 16896420 2024-03-07 15:06:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 15:06:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:06:59 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16896420 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16896420 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 16896420 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-07 15:06:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-07 15:06:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:06:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 15:06:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:06:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2580 try 0 launched on executor multi with executor id 2581 parsl.dataflow.dflow INFO: Standard output for task 2580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/a9419541-1396-469e-b8be-b844bb916090_isrForCrosstalkSources_1125923_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/a9419541-1396-469e-b8be-b844bb916090_isrForCrosstalkSources_1125923_38.stderr parsl.dataflow.dflow DEBUG: Task 2581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2581 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2581 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2581 has memoization hash 3089ab3bb939b9703c8a87fe40c46c16 parsl.dataflow.memoization INFO: Task 2581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2581 try 0 launched on executor multi with executor id 2582 parsl.dataflow.dflow INFO: Standard output for task 2581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/71e783a2-2d18-4dd4-a973-7672c126a6f2_isrForCrosstalkSources_1125907_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/71e783a2-2d18-4dd4-a973-7672c126a6f2_isrForCrosstalkSources_1125907_15.stderr parsl.dataflow.dflow DEBUG: Task 2582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2582 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2582 has memoization hash e36a11c247dd041b9e980aff824afb4a parsl.dataflow.memoization INFO: Task 2582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2582 try 0 launched on executor multi with executor id 2583 parsl.dataflow.dflow INFO: Standard output for task 2582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a306c2e3-5aeb-47b5-a5a2-6c0f4f49f7de_isrForCrosstalkSources_1125900_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a306c2e3-5aeb-47b5-a5a2-6c0f4f49f7de_isrForCrosstalkSources_1125900_10.stderr parsl.dataflow.dflow DEBUG: Task 2583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2583 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2583 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2583 has memoization hash 09aae4996027b84c827ff90144d67fff parsl.dataflow.memoization INFO: Task 2583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39637060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2583 try 0 launched on executor multi with executor id 2584 parsl.dataflow.dflow INFO: Standard output for task 2583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/c5c09f1f-5a5d-4074-b40e-6548b89f91df_isrForCrosstalkSources_1125913_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/c5c09f1f-5a5d-4074-b40e-6548b89f91df_isrForCrosstalkSources_1125913_50.stderr parsl.dataflow.dflow DEBUG: Task 2584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2584 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2584 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2584 has memoization hash 9833f39f14d9cfa940ebb0ec4c313da9 parsl.dataflow.memoization INFO: Task 2584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2584 try 0 launched on executor multi with executor id 2585 parsl.dataflow.dflow INFO: Standard output for task 2584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/c3edee0b-d049-4b1e-acbd-49dfb1a51c91_isrForCrosstalkSources_1125920_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/c3edee0b-d049-4b1e-acbd-49dfb1a51c91_isrForCrosstalkSources_1125920_41.stderr parsl.dataflow.dflow DEBUG: Task 2585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2585 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2585 has memoization hash d0bc3622270fb52db1f3ed2580f68226 parsl.dataflow.memoization INFO: Task 2585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396db060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2585 try 0 launched on executor multi with executor id 2586 parsl.dataflow.dflow INFO: Standard output for task 2585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6d4540de-93b2-4020-a185-58b84f185498_isrForCrosstalkSources_1125927_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6d4540de-93b2-4020-a185-58b84f185498_isrForCrosstalkSources_1125927_36.stderr parsl.dataflow.dflow DEBUG: Task 2586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2586 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2586 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2586 has memoization hash aedf48803bdb9dacf56fd249930ba1b0 parsl.dataflow.memoization INFO: Task 2586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2586 try 0 launched on executor multi with executor id 2587 parsl.dataflow.dflow INFO: Standard output for task 2586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/e82dde5e-9326-44ec-896a-9ced353bae1c_isrForCrosstalkSources_1125941_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/e82dde5e-9326-44ec-896a-9ced353bae1c_isrForCrosstalkSources_1125941_56.stderr parsl.dataflow.dflow DEBUG: Task 2587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2587 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2587 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2587 has memoization hash 164788556c34cbf307f38618dfa3e9fd parsl.dataflow.memoization INFO: Task 2587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2587 try 0 launched on executor multi with executor id 2588 parsl.dataflow.dflow INFO: Standard output for task 2587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/fa9b1fb6-08a5-4cb0-93af-6976cb80cff8_isrForCrosstalkSources_1125906_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/fa9b1fb6-08a5-4cb0-93af-6976cb80cff8_isrForCrosstalkSources_1125906_18.stderr parsl.dataflow.dflow DEBUG: Task 2588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2588 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2588 has memoization hash cbf2dd88e0108cc461b8413f073c1041 parsl.dataflow.memoization INFO: Task 2588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2588 try 0 launched on executor multi with executor id 2589 parsl.dataflow.dflow INFO: Standard output for task 2588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/28ef3fa2-4e4f-4fe1-a286-aa50b1d896a5_isrForCrosstalkSources_1125900_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/28ef3fa2-4e4f-4fe1-a286-aa50b1d896a5_isrForCrosstalkSources_1125900_25.stderr parsl.dataflow.dflow DEBUG: Task 2589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2589 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2589 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2589 has memoization hash 17c3fb3b1845503e5d1c38e9038b3999 parsl.dataflow.memoization INFO: Task 2589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396dbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2589 try 0 launched on executor multi with executor id 2590 parsl.dataflow.dflow INFO: Standard output for task 2589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e34d427a-eae8-43c8-9638-f34b7b877741_isrForCrosstalkSources_1125937_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e34d427a-eae8-43c8-9638-f34b7b877741_isrForCrosstalkSources_1125937_43.stderr parsl.dataflow.dflow DEBUG: Task 2590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2590 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2590 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2590 has memoization hash 46026d6aee7c5c1a93f268556c9abbe0 parsl.dataflow.memoization INFO: Task 2590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2590 try 0 launched on executor multi with executor id 2591 parsl.dataflow.dflow INFO: Standard output for task 2590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/df9b56fb-1d28-4cbc-ab0d-75b3dad48fa2_isrForCrosstalkSources_1125944_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/df9b56fb-1d28-4cbc-ab0d-75b3dad48fa2_isrForCrosstalkSources_1125944_8.stderr parsl.dataflow.dflow DEBUG: Task 2591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2591 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2591 has memoization hash 4ee7843cb6613bddc8d9e1f0cf288114 parsl.dataflow.memoization INFO: Task 2591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396dbec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2591 try 0 launched on executor multi with executor id 2592 parsl.dataflow.dflow INFO: Standard output for task 2591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/46edbd82-7ea3-493e-863f-6e825a844bb9_isrForCrosstalkSources_1125940_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/46edbd82-7ea3-493e-863f-6e825a844bb9_isrForCrosstalkSources_1125940_21.stderr parsl.dataflow.dflow DEBUG: Task 2592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2592 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2592 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2592 has memoization hash 5f6d3530e7b7ffba320310f201327d02 parsl.dataflow.memoization INFO: Task 2592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396dbce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2592 try 0 launched on executor multi with executor id 2593 parsl.dataflow.dflow INFO: Standard output for task 2592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/b078ae62-3f94-4dc2-9160-93c6ac3bc782_isrForCrosstalkSources_1125920_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/b078ae62-3f94-4dc2-9160-93c6ac3bc782_isrForCrosstalkSources_1125920_16.stderr parsl.dataflow.dflow DEBUG: Task 2593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2593 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2593 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2593 has memoization hash c50cc850609b2aea737966863165d86a parsl.dataflow.memoization INFO: Task 2593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396dbd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2593 try 0 launched on executor multi with executor id 2594 parsl.dataflow.dflow INFO: Standard output for task 2593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/a373be8d-5b09-421f-ae6d-cc9761ec849c_isrForCrosstalkSources_1125889_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/a373be8d-5b09-421f-ae6d-cc9761ec849c_isrForCrosstalkSources_1125889_35.stderr parsl.dataflow.dflow DEBUG: Task 2594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2594 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2594 has memoization hash 346c582f101aedae9d9f7c993539c076 parsl.dataflow.memoization INFO: Task 2594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2594 try 0 launched on executor multi with executor id 2595 parsl.dataflow.dflow INFO: Standard output for task 2594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/8ff18580-922c-4e32-b6ec-b14e88ea640a_isrForCrosstalkSources_1125930_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/8ff18580-922c-4e32-b6ec-b14e88ea640a_isrForCrosstalkSources_1125930_6.stderr parsl.dataflow.dflow DEBUG: Task 2595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2595 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2595 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2595 has memoization hash d1fc2a34249253d0909865c0ed2d47dc parsl.dataflow.memoization INFO: Task 2595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2595 try 0 launched on executor multi with executor id 2596 parsl.dataflow.dflow INFO: Standard output for task 2595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/33216f62-04c5-4d41-88a9-b6558bcc4bac_isrForCrosstalkSources_1125931_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/33216f62-04c5-4d41-88a9-b6558bcc4bac_isrForCrosstalkSources_1125931_16.stderr parsl.dataflow.dflow DEBUG: Task 2596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2596 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2596 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2596 has memoization hash e2baef57d1077afe2ae7c6f386be7d27 parsl.dataflow.memoization INFO: Task 2596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2596 try 0 launched on executor multi with executor id 2597 parsl.dataflow.dflow INFO: Standard output for task 2596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/e05f0228-1bac-4f20-8180-28f93bc31c88_isrForCrosstalkSources_1125910_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/e05f0228-1bac-4f20-8180-28f93bc31c88_isrForCrosstalkSources_1125910_12.stderr parsl.dataflow.dflow DEBUG: Task 2597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2597 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2597 has memoization hash 6bd23afb5d34d6d2c2be8fcaf2682723 parsl.dataflow.memoization INFO: Task 2597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2597 try 0 launched on executor multi with executor id 2598 parsl.dataflow.dflow INFO: Standard output for task 2597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c99b7296-4f18-435b-87ea-0ed8c00b1abc_isrForCrosstalkSources_1125893_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c99b7296-4f18-435b-87ea-0ed8c00b1abc_isrForCrosstalkSources_1125893_41.stderr parsl.dataflow.dflow DEBUG: Task 2598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2598 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2598 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2598 has memoization hash bfaed8476dfb82c507a3942028c09de3 parsl.dataflow.memoization INFO: Task 2598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2598 try 0 launched on executor multi with executor id 2599 parsl.dataflow.dflow INFO: Standard output for task 2598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/80d408d3-1aec-4801-bbe0-2dcff0dcd59d_isrForCrosstalkSources_1125913_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/80d408d3-1aec-4801-bbe0-2dcff0dcd59d_isrForCrosstalkSources_1125913_54.stderr parsl.dataflow.dflow DEBUG: Task 2599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2599 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2599 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2599 has memoization hash 971bb8b1339f32bd4a18e99e9899f393 parsl.dataflow.memoization INFO: Task 2599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2599 try 0 launched on executor multi with executor id 2600 parsl.dataflow.dflow INFO: Standard output for task 2599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/41e15ebf-2e37-4319-9994-47770a91dcf1_isrForCrosstalkSources_1125943_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/41e15ebf-2e37-4319-9994-47770a91dcf1_isrForCrosstalkSources_1125943_9.stderr parsl.dataflow.dflow DEBUG: Task 2600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2600 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2600 has memoization hash 4ce9c833d0b8a1572df502358a26c6ed parsl.dataflow.memoization INFO: Task 2600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2600 try 0 launched on executor multi with executor id 2601 parsl.dataflow.dflow INFO: Standard output for task 2600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/0b48e4ee-bde2-4985-94c7-b29b7ea669a2_isrForCrosstalkSources_1125933_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/0b48e4ee-bde2-4985-94c7-b29b7ea669a2_isrForCrosstalkSources_1125933_39.stderr parsl.dataflow.dflow DEBUG: Task 2601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2601 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2601 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2601 has memoization hash 3dd296f837fb46bec61910df87bc23cd parsl.dataflow.memoization INFO: Task 2601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2601 try 0 launched on executor multi with executor id 2602 parsl.dataflow.dflow INFO: Standard output for task 2601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/19acad07-e959-48c8-92f6-74e10a178f8b_isrForCrosstalkSources_1125919_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/19acad07-e959-48c8-92f6-74e10a178f8b_isrForCrosstalkSources_1125919_18.stderr parsl.dataflow.dflow DEBUG: Task 2602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2602 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2602 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2602 has memoization hash 5f2f58b9f9d33b989654d2e74f3eacc4 parsl.dataflow.memoization INFO: Task 2602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2602 try 0 launched on executor multi with executor id 2603 parsl.dataflow.dflow INFO: Standard output for task 2602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/fd5aeb53-413f-4af7-9922-9c63f6d67ddc_isrForCrosstalkSources_1125892_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/fd5aeb53-413f-4af7-9922-9c63f6d67ddc_isrForCrosstalkSources_1125892_61.stderr parsl.dataflow.dflow DEBUG: Task 2603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2603 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2603 has memoization hash 45f287e192636b1567b07eacd4d4db20 parsl.dataflow.memoization INFO: Task 2603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2603 try 0 launched on executor multi with executor id 2604 parsl.dataflow.dflow INFO: Standard output for task 2603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/7a2b785e-ab77-432d-a68a-e09a2a277c50_isrForCrosstalkSources_1125920_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/7a2b785e-ab77-432d-a68a-e09a2a277c50_isrForCrosstalkSources_1125920_46.stderr parsl.dataflow.dflow DEBUG: Task 2604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2604 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2604 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2604 has memoization hash 038179bc21ecf16ac4e4689410ed480d parsl.dataflow.memoization INFO: Task 2604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2604 try 0 launched on executor multi with executor id 2605 parsl.dataflow.dflow INFO: Standard output for task 2604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/582ddf64-4d91-494d-b35e-5228492f2b5a_isrForCrosstalkSources_1125891_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/582ddf64-4d91-494d-b35e-5228492f2b5a_isrForCrosstalkSources_1125891_4.stderr parsl.dataflow.dflow DEBUG: Task 2605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2605 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2605 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2605 has memoization hash d257ae2820d8f0bd1f9d3bb2466f0be1 parsl.dataflow.memoization INFO: Task 2605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2605 try 0 launched on executor multi with executor id 2606 parsl.dataflow.dflow INFO: Standard output for task 2605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/8d6f12d9-a2d0-42e1-8ed9-3adc1a073eae_isrForCrosstalkSources_1125920_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/8d6f12d9-a2d0-42e1-8ed9-3adc1a073eae_isrForCrosstalkSources_1125920_22.stderr parsl.dataflow.dflow DEBUG: Task 2606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2606 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2606 has memoization hash 3e676bf547592995a79a1a95ab4c245e parsl.dataflow.memoization INFO: Task 2606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2606 try 0 launched on executor multi with executor id 2607 parsl.dataflow.dflow INFO: Standard output for task 2606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/cecc248a-c133-4b5d-a586-13422a1fbac7_isrForCrosstalkSources_1125889_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/cecc248a-c133-4b5d-a586-13422a1fbac7_isrForCrosstalkSources_1125889_12.stderr parsl.dataflow.dflow DEBUG: Task 2607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2607 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2607 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2607 has memoization hash 6b7093fc7fcf3a507b8f917ad9b0016a parsl.dataflow.memoization INFO: Task 2607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396dade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2607 try 0 launched on executor multi with executor id 2608 parsl.dataflow.dflow INFO: Standard output for task 2607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/2f1d2b29-c548-45cf-ba36-3284a027075e_isrForCrosstalkSources_1125899_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/2f1d2b29-c548-45cf-ba36-3284a027075e_isrForCrosstalkSources_1125899_50.stderr parsl.dataflow.dflow DEBUG: Task 2608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2608 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2608 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2608 has memoization hash bbc1e7b43193e09e0c5b45361520138f parsl.dataflow.memoization INFO: Task 2608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396dae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2608 try 0 launched on executor multi with executor id 2609 parsl.dataflow.dflow INFO: Standard output for task 2608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c55b1c89-5db8-44d0-826a-816c1dceef34_isrForCrosstalkSources_1125910_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c55b1c89-5db8-44d0-826a-816c1dceef34_isrForCrosstalkSources_1125910_18.stderr parsl.dataflow.dflow DEBUG: Task 2609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2609 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2609 has memoization hash c13b50c8cc319477c324642a884245bc parsl.dataflow.memoization INFO: Task 2609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396daca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:06:59 proc_lsst.multi:170 [INFO] [multi] job_id 16896421 proc_lsst.multi INFO: [multi] job_id 16896421 2024-03-07 15:06:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 15:06:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:06:59 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16896421 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16896421 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 16896421 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-07 15:06:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-07 15:06:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:06:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:06:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:06:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:06:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:06:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:06:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 15:06:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:06:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2609 try 0 launched on executor multi with executor id 2610 parsl.dataflow.dflow INFO: Standard output for task 2609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/c2bdf15b-e9b6-4643-a38e-81de43c9e39d_isrForCrosstalkSources_1125936_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/c2bdf15b-e9b6-4643-a38e-81de43c9e39d_isrForCrosstalkSources_1125936_11.stderr parsl.dataflow.dflow DEBUG: Task 2610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2610 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2610 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2610 has memoization hash 9df27037c7c4a2ba605341c47e2cf69f parsl.dataflow.memoization INFO: Task 2610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396dac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2610 try 0 launched on executor multi with executor id 2611 parsl.dataflow.dflow INFO: Standard output for task 2610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/7e2ba598-e218-44f8-a07c-ab1d251b2e07_isrForCrosstalkSources_1125891_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/7e2ba598-e218-44f8-a07c-ab1d251b2e07_isrForCrosstalkSources_1125891_3.stderr parsl.dataflow.dflow DEBUG: Task 2611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2611 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2611 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2611 has memoization hash c1492f6e63631de2e703d0d40c78cac5 parsl.dataflow.memoization INFO: Task 2611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396dab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2611 try 0 launched on executor multi with executor id 2612 parsl.dataflow.dflow INFO: Standard output for task 2611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/0401ea02-1dd7-4509-a05f-b41db826ad9e_isrForCrosstalkSources_1125933_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/0401ea02-1dd7-4509-a05f-b41db826ad9e_isrForCrosstalkSources_1125933_26.stderr parsl.dataflow.dflow DEBUG: Task 2612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2612 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2612 has memoization hash 9ef5670ab91310415a1b313e4c447e24 parsl.dataflow.memoization INFO: Task 2612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2612 try 0 launched on executor multi with executor id 2613 parsl.dataflow.dflow INFO: Standard output for task 2612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/784673cb-bb3f-4159-a0e5-c182390aeb05_isrForCrosstalkSources_1125893_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/784673cb-bb3f-4159-a0e5-c182390aeb05_isrForCrosstalkSources_1125893_47.stderr parsl.dataflow.dflow DEBUG: Task 2613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2613 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2613 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2613 has memoization hash b903e9363bfac0e5557a3b2f99f7d972 parsl.dataflow.memoization INFO: Task 2613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2613 try 0 launched on executor multi with executor id 2614 parsl.dataflow.dflow INFO: Standard output for task 2613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ac21b8de-fdc4-4d09-b773-897cfddced79_isrForCrosstalkSources_1125894_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ac21b8de-fdc4-4d09-b773-897cfddced79_isrForCrosstalkSources_1125894_38.stderr parsl.dataflow.dflow DEBUG: Task 2614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2614 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2614 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2614 has memoization hash 64139d7f627c57df1716e64465cfcbe8 parsl.dataflow.memoization INFO: Task 2614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2614 try 0 launched on executor multi with executor id 2615 parsl.dataflow.dflow INFO: Standard output for task 2614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d62c1277-db3e-40da-97a9-a087192eeee7_isrForCrosstalkSources_1125923_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d62c1277-db3e-40da-97a9-a087192eeee7_isrForCrosstalkSources_1125923_46.stderr parsl.dataflow.dflow DEBUG: Task 2615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2615 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2615 has memoization hash 49536804d63217c02618e8266cb98df3 parsl.dataflow.memoization INFO: Task 2615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395716c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2615 try 0 launched on executor multi with executor id 2616 parsl.dataflow.dflow INFO: Standard output for task 2615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/d290a5f4-2b13-46b0-82ca-1e37257f403b_isrForCrosstalkSources_1125946_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/d290a5f4-2b13-46b0-82ca-1e37257f403b_isrForCrosstalkSources_1125946_31.stderr parsl.dataflow.dflow DEBUG: Task 2616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2616 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2616 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2616 has memoization hash 7027edbc51dda96ba0dd573659adc570 parsl.dataflow.memoization INFO: Task 2616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2616 try 0 launched on executor multi with executor id 2617 parsl.dataflow.dflow INFO: Standard output for task 2616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/6fe25fdf-99f5-46e0-80d3-149635429eda_isrForCrosstalkSources_1125890_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/6fe25fdf-99f5-46e0-80d3-149635429eda_isrForCrosstalkSources_1125890_29.stderr parsl.dataflow.dflow DEBUG: Task 2617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2617 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2617 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2617 has memoization hash 4883fefed0755004efac0ce5006035a5 parsl.dataflow.memoization INFO: Task 2617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2617 try 0 launched on executor multi with executor id 2618 parsl.dataflow.dflow INFO: Standard output for task 2617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/31f1adfb-23b8-4da8-a904-fe1dbb0c68aa_isrForCrosstalkSources_1125942_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/31f1adfb-23b8-4da8-a904-fe1dbb0c68aa_isrForCrosstalkSources_1125942_36.stderr parsl.dataflow.dflow DEBUG: Task 2618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2618 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2618 has memoization hash d056b77f480f2ddc3418b906b6f1a8db parsl.dataflow.memoization INFO: Task 2618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2618 try 0 launched on executor multi with executor id 2619 parsl.dataflow.dflow INFO: Standard output for task 2618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/5ad8d4c4-2a84-4721-a22b-36401a7d7cd8_isrForCrosstalkSources_1125916_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/5ad8d4c4-2a84-4721-a22b-36401a7d7cd8_isrForCrosstalkSources_1125916_1.stderr parsl.dataflow.dflow DEBUG: Task 2619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2619 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2619 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2619 has memoization hash da998a37ba6974d41dd80d1ed236addf parsl.dataflow.memoization INFO: Task 2619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396da980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2619 try 0 launched on executor multi with executor id 2620 parsl.dataflow.dflow INFO: Standard output for task 2619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/8861d821-f52d-4e96-b5d6-8acf2257c125_isrForCrosstalkSources_1125914_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/8861d821-f52d-4e96-b5d6-8acf2257c125_isrForCrosstalkSources_1125914_54.stderr parsl.dataflow.dflow DEBUG: Task 2620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2620 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2620 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2620 has memoization hash 466aa227796e36209247cf8c6a060521 parsl.dataflow.memoization INFO: Task 2620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396daac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2620 try 0 launched on executor multi with executor id 2621 parsl.dataflow.dflow INFO: Standard output for task 2620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/7dc68754-8277-4234-9cf7-f84d1356c43a_isrForCrosstalkSources_1125903_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/7dc68754-8277-4234-9cf7-f84d1356c43a_isrForCrosstalkSources_1125903_7.stderr parsl.dataflow.dflow DEBUG: Task 2621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2621 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2621 has memoization hash 93e79437c1f5a4a8cad0114ad7f1ad0a parsl.dataflow.memoization INFO: Task 2621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396da840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2621 try 0 launched on executor multi with executor id 2622 parsl.dataflow.dflow INFO: Standard output for task 2621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/d23df871-85ee-41a9-9b77-e5ea0eac087f_isrForCrosstalkSources_1125897_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/d23df871-85ee-41a9-9b77-e5ea0eac087f_isrForCrosstalkSources_1125897_51.stderr parsl.dataflow.dflow DEBUG: Task 2622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2622 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2622 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2622 has memoization hash 3a76b99e0d503b920b9d38090c01c75c parsl.dataflow.memoization INFO: Task 2622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396da700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2622 try 0 launched on executor multi with executor id 2623 parsl.dataflow.dflow INFO: Standard output for task 2622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/ed489381-93de-427c-a576-aae78c4381f0_isrForCrosstalkSources_1125925_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/ed489381-93de-427c-a576-aae78c4381f0_isrForCrosstalkSources_1125925_39.stderr parsl.dataflow.dflow DEBUG: Task 2623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2623 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2623 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2623 has memoization hash 1e4114b93cb11c67de00cb674d7830cb parsl.dataflow.memoization INFO: Task 2623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396da8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2623 try 0 launched on executor multi with executor id 2624 parsl.dataflow.dflow INFO: Standard output for task 2623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/0999f154-d663-4f3f-b174-9490c3a6fc11_isrForCrosstalkSources_1125888_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/0999f154-d663-4f3f-b174-9490c3a6fc11_isrForCrosstalkSources_1125888_37.stderr parsl.dataflow.dflow DEBUG: Task 2624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2624 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2624 has memoization hash a80dd5f274adf07bec90905676857c79 parsl.dataflow.memoization INFO: Task 2624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396daa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2624 try 0 launched on executor multi with executor id 2625 parsl.dataflow.dflow INFO: Standard output for task 2624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/04a6ee66-e2ab-4732-ad37-119a53419c7d_isrForCrosstalkSources_1125915_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/04a6ee66-e2ab-4732-ad37-119a53419c7d_isrForCrosstalkSources_1125915_15.stderr parsl.dataflow.dflow DEBUG: Task 2625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2625 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2625 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2625 has memoization hash af0d6dae07314df226ca5b9d7d43ae2a parsl.dataflow.memoization INFO: Task 2625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396da7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2625 try 0 launched on executor multi with executor id 2626 parsl.dataflow.dflow INFO: Standard output for task 2625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/81917196-c0ce-433c-ab56-cb0115073c15_isrForCrosstalkSources_1125923_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/81917196-c0ce-433c-ab56-cb0115073c15_isrForCrosstalkSources_1125923_17.stderr parsl.dataflow.dflow DEBUG: Task 2626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2626 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2626 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2626 has memoization hash bf7d9406c428f547fc50e3856256ad60 parsl.dataflow.memoization INFO: Task 2626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396da660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2626 try 0 launched on executor multi with executor id 2627 parsl.dataflow.dflow INFO: Standard output for task 2626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/090f92aa-bce2-4fea-893e-3982e9dd36c5_isrForCrosstalkSources_1125935_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/090f92aa-bce2-4fea-893e-3982e9dd36c5_isrForCrosstalkSources_1125935_23.stderr parsl.dataflow.dflow DEBUG: Task 2627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2627 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2627 has memoization hash e8e4cc985b528ab8b02792691a5d0fb6 parsl.dataflow.memoization INFO: Task 2627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396da520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2627 try 0 launched on executor multi with executor id 2628 parsl.dataflow.dflow INFO: Standard output for task 2627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/bac2a3a2-4d9c-42ee-b344-f9cf33313e77_isrForCrosstalkSources_1125902_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/bac2a3a2-4d9c-42ee-b344-f9cf33313e77_isrForCrosstalkSources_1125902_62.stderr parsl.dataflow.dflow DEBUG: Task 2628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2628 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2628 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2628 has memoization hash c78a158b5a03ca0d0d90d2a25f21ea8c parsl.dataflow.memoization INFO: Task 2628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396da480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2628 try 0 launched on executor multi with executor id 2629 parsl.dataflow.dflow INFO: Standard output for task 2628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/7c3ec2f7-788a-45f4-99c9-07fabeafc659_isrForCrosstalkSources_1125933_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/7c3ec2f7-788a-45f4-99c9-07fabeafc659_isrForCrosstalkSources_1125933_27.stderr parsl.dataflow.dflow DEBUG: Task 2629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2629 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2629 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2629 has memoization hash 2a344577211436bf96232875b4dfcb70 parsl.dataflow.memoization INFO: Task 2629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396da3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2629 try 0 launched on executor multi with executor id 2630 parsl.dataflow.dflow INFO: Standard output for task 2629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/27b9a12c-8cd3-47b8-bcb3-82c6ebe4b84e_isrForCrosstalkSources_1125899_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/27b9a12c-8cd3-47b8-bcb3-82c6ebe4b84e_isrForCrosstalkSources_1125899_53.stderr parsl.dataflow.dflow DEBUG: Task 2630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2630 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2630 has memoization hash 6ccd2b474bfa0f58f4f99deaeea5ef58 parsl.dataflow.memoization INFO: Task 2630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396da5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2630 try 0 launched on executor multi with executor id 2631 parsl.dataflow.dflow INFO: Standard output for task 2630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/10715943-520b-445f-b0b4-17cdf4e61320_isrForCrosstalkSources_1125899_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/10715943-520b-445f-b0b4-17cdf4e61320_isrForCrosstalkSources_1125899_5.stderr parsl.dataflow.dflow DEBUG: Task 2631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2631 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2631 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2631 has memoization hash 2f15cc5f556ad918b603bcd6ec99105a parsl.dataflow.memoization INFO: Task 2631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396da340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2631 try 0 launched on executor multi with executor id 2632 parsl.dataflow.dflow INFO: Standard output for task 2631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/b8ec21d3-7af7-4042-9018-d04bd807d3e0_isrForCrosstalkSources_1125914_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/b8ec21d3-7af7-4042-9018-d04bd807d3e0_isrForCrosstalkSources_1125914_52.stderr parsl.dataflow.dflow DEBUG: Task 2632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2632 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2632 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2632 has memoization hash 918b190e6604a655d27cf78f0a3b7a60 parsl.dataflow.memoization INFO: Task 2632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396da200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2632 try 0 launched on executor multi with executor id 2633 parsl.dataflow.dflow INFO: Standard output for task 2632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/905271c9-2e93-40e2-a675-399643d26eea_isrForCrosstalkSources_1125904_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/905271c9-2e93-40e2-a675-399643d26eea_isrForCrosstalkSources_1125904_19.stderr parsl.dataflow.dflow DEBUG: Task 2633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2633 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2633 has memoization hash af6b343fcf7963c36f6d2b324b1d3529 parsl.dataflow.memoization INFO: Task 2633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396da160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2633 try 0 launched on executor multi with executor id 2634 parsl.dataflow.dflow INFO: Standard output for task 2633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/71bd08f7-02c2-4e09-bdc5-298733ca65bf_isrForCrosstalkSources_1125917_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/71bd08f7-02c2-4e09-bdc5-298733ca65bf_isrForCrosstalkSources_1125917_42.stderr parsl.dataflow.dflow DEBUG: Task 2634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2634 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2634 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2634 has memoization hash 5b39706dcf2eb942b91baa77aae67907 parsl.dataflow.memoization INFO: Task 2634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396da2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2634 try 0 launched on executor multi with executor id 2635 parsl.dataflow.dflow INFO: Standard output for task 2634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/1d7e21a8-6acb-4cbc-94ca-e6c8d37df03f_isrForCrosstalkSources_1125889_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/1d7e21a8-6acb-4cbc-94ca-e6c8d37df03f_isrForCrosstalkSources_1125889_55.stderr parsl.dataflow.dflow DEBUG: Task 2635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2635 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2635 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2635 has memoization hash d0e8ad2af913f1747960767347e56ebb parsl.dataflow.memoization INFO: Task 2635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2635 try 0 launched on executor multi with executor id 2636 parsl.dataflow.dflow INFO: Standard output for task 2635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/3808e67a-9e70-4a19-a97c-20df76fb6558_isrForCrosstalkSources_1125925_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/3808e67a-9e70-4a19-a97c-20df76fb6558_isrForCrosstalkSources_1125925_16.stderr parsl.dataflow.dflow DEBUG: Task 2636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2636 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2636 has memoization hash 21387d9ad0e57c0ca589f21371fad52b parsl.dataflow.memoization INFO: Task 2636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2636 try 0 launched on executor multi with executor id 2637 parsl.dataflow.dflow INFO: Standard output for task 2636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/fc7f1af3-170c-4705-981f-c10b65647dc9_isrForCrosstalkSources_1125902_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/fc7f1af3-170c-4705-981f-c10b65647dc9_isrForCrosstalkSources_1125902_56.stderr parsl.dataflow.dflow DEBUG: Task 2637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2637 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2637 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2637 has memoization hash 9d613359a933a69fcc9fda153bf87f80 parsl.dataflow.memoization INFO: Task 2637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396da0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2637 try 0 launched on executor multi with executor id 2638 parsl.dataflow.dflow INFO: Standard output for task 2637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/354aa9b4-46ba-437b-bcbe-f2e986a7ba4c_isrForCrosstalkSources_1125946_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/354aa9b4-46ba-437b-bcbe-f2e986a7ba4c_isrForCrosstalkSources_1125946_34.stderr parsl.dataflow.dflow DEBUG: Task 2638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2638 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2638 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2638 has memoization hash 88cae02fe7f6a884bbc70920c8006efb parsl.dataflow.memoization INFO: Task 2638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2638 try 0 launched on executor multi with executor id 2639 parsl.dataflow.dflow INFO: Standard output for task 2638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/0601d9a0-d2e9-468c-af92-e8f3508221d3_isrForCrosstalkSources_1125929_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/0601d9a0-d2e9-468c-af92-e8f3508221d3_isrForCrosstalkSources_1125929_15.stderr parsl.dataflow.dflow DEBUG: Task 2639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2639 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2639 has memoization hash 66b7f33b7efd04a75f3eca6d71c90a19 parsl.dataflow.memoization INFO: Task 2639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396da020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2639 try 0 launched on executor multi with executor id 2640 parsl.dataflow.dflow INFO: Standard output for task 2639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/1dd39909-b27d-4971-b8f4-f3521f1eeaa5_isrForCrosstalkSources_1125899_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/1dd39909-b27d-4971-b8f4-f3521f1eeaa5_isrForCrosstalkSources_1125899_48.stderr parsl.dataflow.dflow DEBUG: Task 2640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2640 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2640 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2640 has memoization hash 98348b458a87065ca5a7527b3ecc73a7 parsl.dataflow.memoization INFO: Task 2640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2640 try 0 launched on executor multi with executor id 2641 parsl.dataflow.dflow INFO: Standard output for task 2640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/8cfd3d42-4d38-4cfb-ab1c-8879c8264e3b_isrForCrosstalkSources_1125941_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/8cfd3d42-4d38-4cfb-ab1c-8879c8264e3b_isrForCrosstalkSources_1125941_54.stderr parsl.dataflow.dflow DEBUG: Task 2641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2641 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2641 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2641 has memoization hash 0154df8b4f43d037ab65edb0e4b5f43b parsl.dataflow.memoization INFO: Task 2641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2641 try 0 launched on executor multi with executor id 2642 parsl.dataflow.dflow INFO: Standard output for task 2641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/0355559d-91c6-4655-bd54-5c1cd127038d_isrForCrosstalkSources_1125906_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/0355559d-91c6-4655-bd54-5c1cd127038d_isrForCrosstalkSources_1125906_39.stderr parsl.dataflow.dflow DEBUG: Task 2642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2642 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2642 has memoization hash 3e000df5f47c8daeaf3e39cd21dbe484 parsl.dataflow.memoization INFO: Task 2642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2642 try 0 launched on executor multi with executor id 2643 parsl.dataflow.dflow INFO: Standard output for task 2642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/200f23f1-7c27-45d7-9843-ee6575022775_isrForCrosstalkSources_1125910_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/200f23f1-7c27-45d7-9843-ee6575022775_isrForCrosstalkSources_1125910_58.stderr parsl.dataflow.dflow DEBUG: Task 2643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2643 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2643 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2643 has memoization hash bb9df3de9355be82db882453470bd4b2 parsl.dataflow.memoization INFO: Task 2643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396db1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2643 try 0 launched on executor multi with executor id 2644 parsl.dataflow.dflow INFO: Standard output for task 2643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/18efb0da-f470-49e0-a386-66d44fed0733_isrForCrosstalkSources_1125898_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/18efb0da-f470-49e0-a386-66d44fed0733_isrForCrosstalkSources_1125898_51.stderr parsl.dataflow.dflow DEBUG: Task 2644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2644 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2644 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2644 has memoization hash ae31c9ffba05d4e824443524a44a299a parsl.dataflow.memoization INFO: Task 2644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396db240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2644 try 0 launched on executor multi with executor id 2645 parsl.dataflow.dflow INFO: Standard output for task 2644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/338ceecf-4c2e-4ccd-a54c-0515f7bb2fca_isrForCrosstalkSources_1125901_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/338ceecf-4c2e-4ccd-a54c-0515f7bb2fca_isrForCrosstalkSources_1125901_37.stderr parsl.dataflow.dflow DEBUG: Task 2645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2645 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2645 has memoization hash 8c58906601c92e4c293992d67add0bb0 parsl.dataflow.memoization INFO: Task 2645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396db2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2645 try 0 launched on executor multi with executor id 2646 parsl.dataflow.dflow INFO: Standard output for task 2645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/d2d8f177-b399-4da0-8cbb-6c62b590a90d_isrForCrosstalkSources_1125906_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/d2d8f177-b399-4da0-8cbb-6c62b590a90d_isrForCrosstalkSources_1125906_61.stderr parsl.dataflow.dflow DEBUG: Task 2646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2646 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2646 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2646 has memoization hash 11090402378d1869aa334c0dc75f7dda parsl.dataflow.memoization INFO: Task 2646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396db380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2646 try 0 launched on executor multi with executor id 2647 parsl.dataflow.dflow INFO: Standard output for task 2646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/d7bd947d-2f7c-40e4-a4d9-f14638c9ceec_isrForCrosstalkSources_1125929_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/d7bd947d-2f7c-40e4-a4d9-f14638c9ceec_isrForCrosstalkSources_1125929_13.stderr parsl.dataflow.dflow DEBUG: Task 2647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2647 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2647 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2647 has memoization hash b1e90fca251b55bb60388be0922511de parsl.dataflow.memoization INFO: Task 2647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396db100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:00 proc_lsst.multi:170 [INFO] [multi] job_id 16896422 proc_lsst.multi INFO: [multi] job_id 16896422 2024-03-07 15:07:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 15:07:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:00 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16896422 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16896422 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 16896422 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-07 15:07:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-07 15:07:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 15:07:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2647 try 0 launched on executor multi with executor id 2648 parsl.dataflow.dflow INFO: Standard output for task 2647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/4185f51e-72bd-439f-925e-7aced67f193d_isrForCrosstalkSources_1125946_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/4185f51e-72bd-439f-925e-7aced67f193d_isrForCrosstalkSources_1125946_28.stderr parsl.dataflow.dflow DEBUG: Task 2648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2648 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2648 has memoization hash d16027d98958e09b2a9e12456e940e7a parsl.dataflow.memoization INFO: Task 2648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3d66d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2648 try 0 launched on executor multi with executor id 2649 parsl.dataflow.dflow INFO: Standard output for task 2648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d718149f-f134-4c06-9cab-eec31ed795e6_isrForCrosstalkSources_1125940_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d718149f-f134-4c06-9cab-eec31ed795e6_isrForCrosstalkSources_1125940_36.stderr parsl.dataflow.dflow DEBUG: Task 2649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2649 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2649 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2649 has memoization hash 5291f7b8c837625c63c960644b35ab0d parsl.dataflow.memoization INFO: Task 2649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39bc47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2649 try 0 launched on executor multi with executor id 2650 parsl.dataflow.dflow INFO: Standard output for task 2649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/5d26cb69-64b9-40c1-83ee-ff6fcf1d502e_isrForCrosstalkSources_1125912_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/5d26cb69-64b9-40c1-83ee-ff6fcf1d502e_isrForCrosstalkSources_1125912_58.stderr parsl.dataflow.dflow DEBUG: Task 2650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2650 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2650 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2650 has memoization hash 23519667df16c2deb663f2d9621d10d5 parsl.dataflow.memoization INFO: Task 2650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396dbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2650 try 0 launched on executor multi with executor id 2651 parsl.dataflow.dflow INFO: Standard output for task 2650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/152f8d47-d4fe-4882-9835-147a730bd23a_isrForCrosstalkSources_1125928_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/152f8d47-d4fe-4882-9835-147a730bd23a_isrForCrosstalkSources_1125928_3.stderr parsl.dataflow.dflow DEBUG: Task 2651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2651 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2651 has memoization hash fa35f58653d92aba3825bebab0b7ee94 parsl.dataflow.memoization INFO: Task 2651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396d8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2651 try 0 launched on executor multi with executor id 2652 parsl.dataflow.dflow INFO: Standard output for task 2651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/70d48dac-c112-429d-bdc1-b19e56ed4150_isrForCrosstalkSources_1125889_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/70d48dac-c112-429d-bdc1-b19e56ed4150_isrForCrosstalkSources_1125889_13.stderr parsl.dataflow.dflow DEBUG: Task 2652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2652 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2652 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2652 has memoization hash 685792b34fdec63fa0e3b9a369a24972 parsl.dataflow.memoization INFO: Task 2652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a397e3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2652 try 0 launched on executor multi with executor id 2653 parsl.dataflow.dflow INFO: Standard output for task 2652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/602089ad-50f1-45b3-aa35-7d53d998a8a5_isrForCrosstalkSources_1125931_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/602089ad-50f1-45b3-aa35-7d53d998a8a5_isrForCrosstalkSources_1125931_52.stderr parsl.dataflow.dflow DEBUG: Task 2653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2653 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2653 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2653 has memoization hash e839246eaff62bc5808c4b20661b9eba parsl.dataflow.memoization INFO: Task 2653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396dbba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2653 try 0 launched on executor multi with executor id 2654 parsl.dataflow.dflow INFO: Standard output for task 2653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/1305bab8-5379-4bc0-b2e4-2ba893ce9851_isrForCrosstalkSources_1125922_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/1305bab8-5379-4bc0-b2e4-2ba893ce9851_isrForCrosstalkSources_1125922_54.stderr parsl.dataflow.dflow DEBUG: Task 2654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2654 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2654 has memoization hash 3ac301291ded531f69c6f6d861f27a3e parsl.dataflow.memoization INFO: Task 2654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395731a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2654 try 0 launched on executor multi with executor id 2655 parsl.dataflow.dflow INFO: Standard output for task 2654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f01363e1-2d93-4f7a-9f79-fe9ac81f5a55_isrForCrosstalkSources_1125930_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f01363e1-2d93-4f7a-9f79-fe9ac81f5a55_isrForCrosstalkSources_1125930_9.stderr parsl.dataflow.dflow DEBUG: Task 2655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2655 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2655 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2655 has memoization hash ef1463f95ba52f677d15c793c849d64f parsl.dataflow.memoization INFO: Task 2655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2655 try 0 launched on executor multi with executor id 2656 parsl.dataflow.dflow INFO: Standard output for task 2655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/404b9a87-0d86-42ee-8830-90b5b741fee4_isrForCrosstalkSources_1125946_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/404b9a87-0d86-42ee-8830-90b5b741fee4_isrForCrosstalkSources_1125946_62.stderr parsl.dataflow.dflow DEBUG: Task 2656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2656 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2656 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2656 has memoization hash 8f36f1ec10f221abbff0bf9c80dec04e parsl.dataflow.memoization INFO: Task 2656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2656 try 0 launched on executor multi with executor id 2657 parsl.dataflow.dflow INFO: Standard output for task 2656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/677d2168-ea92-441e-9287-7358ea1e355b_isrForCrosstalkSources_1125910_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/677d2168-ea92-441e-9287-7358ea1e355b_isrForCrosstalkSources_1125910_26.stderr parsl.dataflow.dflow DEBUG: Task 2657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2657 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2657 has memoization hash f1364b7bcfde5a230dd360fb961d5c3c parsl.dataflow.memoization INFO: Task 2657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2657 try 0 launched on executor multi with executor id 2658 parsl.dataflow.dflow INFO: Standard output for task 2657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/f23d6fec-350e-4c8a-84e1-8dfc87cad1da_isrForCrosstalkSources_1125893_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/f23d6fec-350e-4c8a-84e1-8dfc87cad1da_isrForCrosstalkSources_1125893_38.stderr parsl.dataflow.dflow DEBUG: Task 2658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2658 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2658 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2658 has memoization hash 4f3601826092331db0f6fd2d33a8cdab parsl.dataflow.memoization INFO: Task 2658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396db9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2658 try 0 launched on executor multi with executor id 2659 parsl.dataflow.dflow INFO: Standard output for task 2658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/8774bc01-e0d8-4bb3-9ebd-b8dd643878ad_isrForCrosstalkSources_1125904_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/8774bc01-e0d8-4bb3-9ebd-b8dd643878ad_isrForCrosstalkSources_1125904_25.stderr parsl.dataflow.dflow DEBUG: Task 2659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2659 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2659 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2659 has memoization hash 6bbb7694ac35e35961df572ce348f5ba parsl.dataflow.memoization INFO: Task 2659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396dbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2659 try 0 launched on executor multi with executor id 2660 parsl.dataflow.dflow INFO: Standard output for task 2659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/844986aa-a028-4ab6-bbe3-54d7ec354097_isrForCrosstalkSources_1125913_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/844986aa-a028-4ab6-bbe3-54d7ec354097_isrForCrosstalkSources_1125913_28.stderr parsl.dataflow.dflow DEBUG: Task 2660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2660 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2660 has memoization hash 0d2ccb5df92fcd0507addcbba2145acd parsl.dataflow.memoization INFO: Task 2660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396db920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2660 try 0 launched on executor multi with executor id 2661 parsl.dataflow.dflow INFO: Standard output for task 2660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/f4f11d09-3ce5-44d3-9643-7620d9f2cf99_isrForCrosstalkSources_1125901_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/f4f11d09-3ce5-44d3-9643-7620d9f2cf99_isrForCrosstalkSources_1125901_25.stderr parsl.dataflow.dflow DEBUG: Task 2661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2661 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2661 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2661 has memoization hash a4cdccdd20dce537098211273d7012fc parsl.dataflow.memoization INFO: Task 2661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396db600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2661 try 0 launched on executor multi with executor id 2662 parsl.dataflow.dflow INFO: Standard output for task 2661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6cced87c-8688-4cd5-957d-a66b38896fae_isrForCrosstalkSources_1125909_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6cced87c-8688-4cd5-957d-a66b38896fae_isrForCrosstalkSources_1125909_21.stderr parsl.dataflow.dflow DEBUG: Task 2662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2662 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2662 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2662 has memoization hash d39f44926355b3c25b5116996e89f83e parsl.dataflow.memoization INFO: Task 2662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396db6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2662 try 0 launched on executor multi with executor id 2663 parsl.dataflow.dflow INFO: Standard output for task 2662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/78887e37-cbb9-48a9-9252-ece739ab110f_isrForCrosstalkSources_1125917_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/78887e37-cbb9-48a9-9252-ece739ab110f_isrForCrosstalkSources_1125917_50.stderr parsl.dataflow.dflow DEBUG: Task 2663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2663 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2663 has memoization hash a034fe4f82902ad1f04de63b6cc30110 parsl.dataflow.memoization INFO: Task 2663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396db4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2663 try 0 launched on executor multi with executor id 2664 parsl.dataflow.dflow INFO: Standard output for task 2663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/54129f26-6071-47dc-8280-d0d5926e9784_isrForCrosstalkSources_1125933_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/54129f26-6071-47dc-8280-d0d5926e9784_isrForCrosstalkSources_1125933_24.stderr parsl.dataflow.dflow DEBUG: Task 2664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2664 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2664 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2664 has memoization hash 7589885573be367721c3e09759094e83 parsl.dataflow.memoization INFO: Task 2664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396db7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2664 try 0 launched on executor multi with executor id 2665 parsl.dataflow.dflow INFO: Standard output for task 2664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/aa5164a1-721d-4d42-bffa-1e798bef8c43_isrForCrosstalkSources_1125934_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/aa5164a1-721d-4d42-bffa-1e798bef8c43_isrForCrosstalkSources_1125934_39.stderr parsl.dataflow.dflow DEBUG: Task 2665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2665 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2665 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2665 has memoization hash 4848e99f7744ee28824687bccc488b98 parsl.dataflow.memoization INFO: Task 2665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396db880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2665 try 0 launched on executor multi with executor id 2666 parsl.dataflow.dflow INFO: Standard output for task 2665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/fe63a309-6980-4a8d-80bc-40d108dfc1db_isrForCrosstalkSources_1125904_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/fe63a309-6980-4a8d-80bc-40d108dfc1db_isrForCrosstalkSources_1125904_32.stderr parsl.dataflow.dflow DEBUG: Task 2666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2666 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2666 has memoization hash 68aed095f428a7982e95e590da507a00 parsl.dataflow.memoization INFO: Task 2666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2666 try 0 launched on executor multi with executor id 2667 parsl.dataflow.dflow INFO: Standard output for task 2666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/87a0764e-f347-4a78-81f3-a0351e5d22a2_isrForCrosstalkSources_1125918_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/87a0764e-f347-4a78-81f3-a0351e5d22a2_isrForCrosstalkSources_1125918_34.stderr parsl.dataflow.dflow DEBUG: Task 2667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2667 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2667 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2667 has memoization hash 4f7b9c14e9bc7dc16194dbf03410e38d parsl.dataflow.memoization INFO: Task 2667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395713a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2667 try 0 launched on executor multi with executor id 2668 parsl.dataflow.dflow INFO: Standard output for task 2667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/5aa2fb4d-e3a1-4c23-bf6b-1bfa2ddf9553_isrForCrosstalkSources_1125938_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/5aa2fb4d-e3a1-4c23-bf6b-1bfa2ddf9553_isrForCrosstalkSources_1125938_27.stderr parsl.dataflow.dflow DEBUG: Task 2668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2668 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2668 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2668 has memoization hash 06207e1a43bc85343c021b8ea7f7f8a7 parsl.dataflow.memoization INFO: Task 2668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2668 try 0 launched on executor multi with executor id 2669 parsl.dataflow.dflow INFO: Standard output for task 2668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/d5828e4d-66e4-4a94-b45b-fb4fb9ee81ac_isrForCrosstalkSources_1125928_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/d5828e4d-66e4-4a94-b45b-fb4fb9ee81ac_isrForCrosstalkSources_1125928_5.stderr parsl.dataflow.dflow DEBUG: Task 2669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2669 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2669 has memoization hash e48a421f4aa8a991ea4e833ef43f8546 parsl.dataflow.memoization INFO: Task 2669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395700e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2669 try 0 launched on executor multi with executor id 2670 parsl.dataflow.dflow INFO: Standard output for task 2669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/f5ddeb07-c8e1-4273-bf8e-fe1d89be22b5_isrForCrosstalkSources_1125921_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/f5ddeb07-c8e1-4273-bf8e-fe1d89be22b5_isrForCrosstalkSources_1125921_31.stderr parsl.dataflow.dflow DEBUG: Task 2670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2670 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2670 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2670 has memoization hash bfa108f9273c284f72b2a591546a1341 parsl.dataflow.memoization INFO: Task 2670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2670 try 0 launched on executor multi with executor id 2671 parsl.dataflow.dflow INFO: Standard output for task 2670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a0ea32da-4ed6-41e1-8c62-98f0af8b475d_isrForCrosstalkSources_1125945_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a0ea32da-4ed6-41e1-8c62-98f0af8b475d_isrForCrosstalkSources_1125945_18.stderr parsl.dataflow.dflow DEBUG: Task 2671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2671 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2671 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2671 has memoization hash 8a3235eb9a21bda2fa029d9843d54507 parsl.dataflow.memoization INFO: Task 2671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395714e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2671 try 0 launched on executor multi with executor id 2672 parsl.dataflow.dflow INFO: Standard output for task 2671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/125496ec-3a41-4fb2-ba23-cbe4dec5d8f7_isrForCrosstalkSources_1125922_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/125496ec-3a41-4fb2-ba23-cbe4dec5d8f7_isrForCrosstalkSources_1125922_24.stderr parsl.dataflow.dflow DEBUG: Task 2672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2672 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2672 has memoization hash cf538658b61da0f7f5bd6428c7030d6b parsl.dataflow.memoization INFO: Task 2672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2672 try 0 launched on executor multi with executor id 2673 parsl.dataflow.dflow INFO: Standard output for task 2672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/0a0ba096-495d-42d1-931f-6ebde9eb2004_isrForCrosstalkSources_1125898_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/0a0ba096-495d-42d1-931f-6ebde9eb2004_isrForCrosstalkSources_1125898_8.stderr parsl.dataflow.dflow DEBUG: Task 2673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2673 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2673 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2673 has memoization hash da38a90291089274eb8ffb7825569f81 parsl.dataflow.memoization INFO: Task 2673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2673 try 0 launched on executor multi with executor id 2674 parsl.dataflow.dflow INFO: Standard output for task 2673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/80c968c4-0ea3-496e-9167-287c81a3eb04_isrForCrosstalkSources_1125926_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/80c968c4-0ea3-496e-9167-287c81a3eb04_isrForCrosstalkSources_1125926_22.stderr parsl.dataflow.dflow DEBUG: Task 2674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2674 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2674 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2674 has memoization hash b743617962f9ef3a67c0899defaf0ad3 parsl.dataflow.memoization INFO: Task 2674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395702c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2674 try 0 launched on executor multi with executor id 2675 parsl.dataflow.dflow INFO: Standard output for task 2674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/ee03c364-ae07-41b8-8085-f1cf73ea089a_isrForCrosstalkSources_1125944_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/ee03c364-ae07-41b8-8085-f1cf73ea089a_isrForCrosstalkSources_1125944_17.stderr parsl.dataflow.dflow DEBUG: Task 2675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2675 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2675 has memoization hash 0af8248d4971079ab57e04b9a959d403 parsl.dataflow.memoization INFO: Task 2675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395704a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2675 try 0 launched on executor multi with executor id 2676 parsl.dataflow.dflow INFO: Standard output for task 2675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/fa149316-25e5-4fb4-8e4c-284c162858b8_isrForCrosstalkSources_1125910_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/fa149316-25e5-4fb4-8e4c-284c162858b8_isrForCrosstalkSources_1125910_39.stderr parsl.dataflow.dflow DEBUG: Task 2676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2676 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2676 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2676 has memoization hash d954be0b61a721d96e04aff15d53d2e2 parsl.dataflow.memoization INFO: Task 2676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395718a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2676 try 0 launched on executor multi with executor id 2677 parsl.dataflow.dflow INFO: Standard output for task 2676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/e2c4e2b2-16dd-4e5c-9b45-6c1730e1837c_isrForCrosstalkSources_1125920_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/e2c4e2b2-16dd-4e5c-9b45-6c1730e1837c_isrForCrosstalkSources_1125920_18.stderr parsl.dataflow.dflow DEBUG: Task 2677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2677 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2677 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2677 has memoization hash cd844ad159431cc8945951cd7571deac parsl.dataflow.memoization INFO: Task 2677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2677 try 0 launched on executor multi with executor id 2678 parsl.dataflow.dflow INFO: Standard output for task 2677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6ad3345d-87bc-4a96-b8e0-fe8c7b1701cd_isrForCrosstalkSources_1125927_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6ad3345d-87bc-4a96-b8e0-fe8c7b1701cd_isrForCrosstalkSources_1125927_34.stderr parsl.dataflow.dflow DEBUG: Task 2678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2678 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2678 has memoization hash a93df66b5a6137e8aaa42f4b5086708d parsl.dataflow.memoization INFO: Task 2678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2678 try 0 launched on executor multi with executor id 2679 parsl.dataflow.dflow INFO: Standard output for task 2678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/7d4a7222-0301-48b2-b58f-fb2f85f810ea_isrForCrosstalkSources_1125920_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/7d4a7222-0301-48b2-b58f-fb2f85f810ea_isrForCrosstalkSources_1125920_38.stderr parsl.dataflow.dflow DEBUG: Task 2679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2679 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2679 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2679 has memoization hash def429cb13d3882680276d2a35716b4b parsl.dataflow.memoization INFO: Task 2679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2679 try 0 launched on executor multi with executor id 2680 parsl.dataflow.dflow INFO: Standard output for task 2679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/91a59eba-d448-4e67-86db-e9478a94c340_isrForCrosstalkSources_1125893_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/91a59eba-d448-4e67-86db-e9478a94c340_isrForCrosstalkSources_1125893_49.stderr parsl.dataflow.dflow DEBUG: Task 2680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2680 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2680 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2680 has memoization hash 896541510877931d2fc88c7eece55d97 parsl.dataflow.memoization INFO: Task 2680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395711c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2680 try 0 launched on executor multi with executor id 2681 parsl.dataflow.dflow INFO: Standard output for task 2680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/44799503-75a1-4145-ace5-3440568aac63_isrForCrosstalkSources_1125926_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/44799503-75a1-4145-ace5-3440568aac63_isrForCrosstalkSources_1125926_31.stderr parsl.dataflow.dflow DEBUG: Task 2681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2681 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2681 has memoization hash 4d42e0f04b889218cc7aef0c561d4f13 parsl.dataflow.memoization INFO: Task 2681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2681 try 0 launched on executor multi with executor id 2682 parsl.dataflow.dflow INFO: Standard output for task 2681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/75222876-dcb8-4bed-af33-18dfdf633718_isrForCrosstalkSources_1125934_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/75222876-dcb8-4bed-af33-18dfdf633718_isrForCrosstalkSources_1125934_50.stderr parsl.dataflow.dflow DEBUG: Task 2682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2682 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2682 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2682 has memoization hash 9a441b12e89bccf8d5fb8a9a0f53cdba parsl.dataflow.memoization INFO: Task 2682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2682 try 0 launched on executor multi with executor id 2683 parsl.dataflow.dflow INFO: Standard output for task 2682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/4ecd1ce1-8879-4f0a-93d1-e899c0555e3c_isrForCrosstalkSources_1125899_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/4ecd1ce1-8879-4f0a-93d1-e899c0555e3c_isrForCrosstalkSources_1125899_52.stderr parsl.dataflow.dflow DEBUG: Task 2683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2683 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2683 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2683 has memoization hash c7abb39c31988e9d10bac35f78e553a4 parsl.dataflow.memoization INFO: Task 2683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2683 try 0 launched on executor multi with executor id 2684 parsl.dataflow.dflow INFO: Standard output for task 2683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/8f1a353c-e40a-4425-9e08-8fdd9c99b55a_isrForCrosstalkSources_1125947_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/8f1a353c-e40a-4425-9e08-8fdd9c99b55a_isrForCrosstalkSources_1125947_47.stderr parsl.dataflow.dflow DEBUG: Task 2684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2684 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2684 has memoization hash b51a9af396b5888d46b69d940312617b parsl.dataflow.memoization INFO: Task 2684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2684 try 0 launched on executor multi with executor id 2685 parsl.dataflow.dflow INFO: Standard output for task 2684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/1c5b1583-1f61-4e07-8410-3d84d0b3266a_isrForCrosstalkSources_1125938_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/1c5b1583-1f61-4e07-8410-3d84d0b3266a_isrForCrosstalkSources_1125938_57.stderr parsl.dataflow.dflow DEBUG: Task 2685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2685 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2685 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2685 has memoization hash 403560196897b13d1c84ac50b9d88cc9 parsl.dataflow.memoization INFO: Task 2685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2685 try 0 launched on executor multi with executor id 2686 parsl.dataflow.dflow INFO: Standard output for task 2685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/76083684-89aa-4576-a31a-b2c95988aadd_isrForCrosstalkSources_1125889_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/76083684-89aa-4576-a31a-b2c95988aadd_isrForCrosstalkSources_1125889_46.stderr parsl.dataflow.dflow DEBUG: Task 2686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2686 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2686 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2686 has memoization hash 3980e6d60bc7382d1b507171bfa696da parsl.dataflow.memoization INFO: Task 2686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2686 try 0 launched on executor multi with executor id 2687 parsl.dataflow.dflow INFO: Standard output for task 2686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/6641d0a6-01a6-496f-a04e-0f6ee4b18b1a_isrForCrosstalkSources_1125906_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/6641d0a6-01a6-496f-a04e-0f6ee4b18b1a_isrForCrosstalkSources_1125906_15.stderr parsl.dataflow.dflow DEBUG: Task 2687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2687 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2687 has memoization hash 76c6cc78ae44bc22d3b92fffb1536b82 parsl.dataflow.memoization INFO: Task 2687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:00 proc_lsst.multi:170 [INFO] [multi] job_id 16896423 proc_lsst.multi INFO: [multi] job_id 16896423 2024-03-07 15:07:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:00 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16896423 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16896423 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 16896423 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-07 15:07:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-07 15:07:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 15:07:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2687 try 0 launched on executor multi with executor id 2688 parsl.dataflow.dflow INFO: Standard output for task 2687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/7659c1fc-c72f-4645-85ec-00f216602468_isrForCrosstalkSources_1125888_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/7659c1fc-c72f-4645-85ec-00f216602468_isrForCrosstalkSources_1125888_23.stderr parsl.dataflow.dflow DEBUG: Task 2688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2688 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2688 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2688 has memoization hash 1a0cf955c8c70499fa8227f815551982 parsl.dataflow.memoization INFO: Task 2688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396dbc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2688 try 0 launched on executor multi with executor id 2689 parsl.dataflow.dflow INFO: Standard output for task 2688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9a67d732-6a7e-4e5e-bbae-2af4ed160398_isrForCrosstalkSources_1125934_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9a67d732-6a7e-4e5e-bbae-2af4ed160398_isrForCrosstalkSources_1125934_18.stderr parsl.dataflow.dflow DEBUG: Task 2689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2689 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2689 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2689 has memoization hash 7318583362a6ff6a1d0a155029191ae2 parsl.dataflow.memoization INFO: Task 2689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396db740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2689 try 0 launched on executor multi with executor id 2690 parsl.dataflow.dflow INFO: Standard output for task 2689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/04c6bb87-387d-4807-8483-0349ccec9678_isrForCrosstalkSources_1125915_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/04c6bb87-387d-4807-8483-0349ccec9678_isrForCrosstalkSources_1125915_30.stderr parsl.dataflow.dflow DEBUG: Task 2690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2690 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2690 has memoization hash eafe8a9216474ca773e00410a506422c parsl.dataflow.memoization INFO: Task 2690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396db560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2690 try 0 launched on executor multi with executor id 2691 parsl.dataflow.dflow INFO: Standard output for task 2690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/62f99ea7-5386-42c3-ae39-487d49e3a082_isrForCrosstalkSources_1125938_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/62f99ea7-5386-42c3-ae39-487d49e3a082_isrForCrosstalkSources_1125938_17.stderr parsl.dataflow.dflow DEBUG: Task 2691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2691 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2691 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2691 has memoization hash 14d76b0d52621ee9ed01741cbf1fcf69 parsl.dataflow.memoization INFO: Task 2691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2691 try 0 launched on executor multi with executor id 2692 parsl.dataflow.dflow INFO: Standard output for task 2691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/0fcd87db-0754-427d-9b20-2195d820c2d0_isrForCrosstalkSources_1125891_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/0fcd87db-0754-427d-9b20-2195d820c2d0_isrForCrosstalkSources_1125891_23.stderr parsl.dataflow.dflow DEBUG: Task 2692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2692 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2692 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2692 has memoization hash 399630f1f13035f8164643ba33389e7b parsl.dataflow.memoization INFO: Task 2692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2692 try 0 launched on executor multi with executor id 2693 parsl.dataflow.dflow INFO: Standard output for task 2692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/47663458-4120-426f-aeca-5103152ca331_isrForCrosstalkSources_1125947_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/47663458-4120-426f-aeca-5103152ca331_isrForCrosstalkSources_1125947_45.stderr parsl.dataflow.dflow DEBUG: Task 2693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2693 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2693 has memoization hash 62d848abc39e7e1f10aa2fb97884731c parsl.dataflow.memoization INFO: Task 2693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2693 try 0 launched on executor multi with executor id 2694 parsl.dataflow.dflow INFO: Standard output for task 2693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/39489c5d-9db3-4f98-a089-da9b0a3939f1_isrForCrosstalkSources_1125920_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/39489c5d-9db3-4f98-a089-da9b0a3939f1_isrForCrosstalkSources_1125920_12.stderr parsl.dataflow.dflow DEBUG: Task 2694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2694 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2694 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2694 has memoization hash 4e207c50ab9854e7dafdc861afd36282 parsl.dataflow.memoization INFO: Task 2694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2694 try 0 launched on executor multi with executor id 2695 parsl.dataflow.dflow INFO: Standard output for task 2694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/297ddcad-9888-43b8-8ce6-c068861f18f6_isrForCrosstalkSources_1125943_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/297ddcad-9888-43b8-8ce6-c068861f18f6_isrForCrosstalkSources_1125943_7.stderr parsl.dataflow.dflow DEBUG: Task 2695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2695 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2695 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2695 has memoization hash cae42d4946081855dbe5b0551192bacb parsl.dataflow.memoization INFO: Task 2695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2695 try 0 launched on executor multi with executor id 2696 parsl.dataflow.dflow INFO: Standard output for task 2695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/ceb601ad-0674-4c60-baf7-f06e38891396_isrForCrosstalkSources_1125919_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/ceb601ad-0674-4c60-baf7-f06e38891396_isrForCrosstalkSources_1125919_7.stderr parsl.dataflow.dflow DEBUG: Task 2696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2696 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2696 has memoization hash dd379f2e0511784e01646eb959aa1a4f parsl.dataflow.memoization INFO: Task 2696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2696 try 0 launched on executor multi with executor id 2697 parsl.dataflow.dflow INFO: Standard output for task 2696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/79de3fa9-1fd4-4ccc-9ba4-851c566ab04e_isrForCrosstalkSources_1125906_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/79de3fa9-1fd4-4ccc-9ba4-851c566ab04e_isrForCrosstalkSources_1125906_2.stderr parsl.dataflow.dflow DEBUG: Task 2697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2697 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2697 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2697 has memoization hash c8c8551d8cfbe00c31b7576342c4a005 parsl.dataflow.memoization INFO: Task 2697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2697 try 0 launched on executor multi with executor id 2698 parsl.dataflow.dflow INFO: Standard output for task 2697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/dec4254c-f626-426f-95db-07cc29f9c31f_isrForCrosstalkSources_1125911_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/dec4254c-f626-426f-95db-07cc29f9c31f_isrForCrosstalkSources_1125911_60.stderr parsl.dataflow.dflow DEBUG: Task 2698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2698 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2698 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2698 has memoization hash edc40e57502d448ac66b31f2ddb0d436 parsl.dataflow.memoization INFO: Task 2698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2698 try 0 launched on executor multi with executor id 2699 parsl.dataflow.dflow INFO: Standard output for task 2698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/57e32e71-6d86-4516-9d42-9755896c6e0e_isrForCrosstalkSources_1125932_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/57e32e71-6d86-4516-9d42-9755896c6e0e_isrForCrosstalkSources_1125932_50.stderr parsl.dataflow.dflow DEBUG: Task 2699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2699 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2699 has memoization hash 2fe629f775734cc3d07442db7a598092 parsl.dataflow.memoization INFO: Task 2699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2699 try 0 launched on executor multi with executor id 2700 parsl.dataflow.dflow INFO: Standard output for task 2699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/c6a23d5f-7e36-45d8-84dd-8cabf942c7e4_isrForCrosstalkSources_1125925_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/c6a23d5f-7e36-45d8-84dd-8cabf942c7e4_isrForCrosstalkSources_1125925_19.stderr parsl.dataflow.dflow DEBUG: Task 2700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2700 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2700 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2700 has memoization hash 724bdc3bb03467c132f35b81a86c1c6b parsl.dataflow.memoization INFO: Task 2700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2700 try 0 launched on executor multi with executor id 2701 parsl.dataflow.dflow INFO: Standard output for task 2700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/c299e983-fefe-4d05-8e33-f8f318974e7d_isrForCrosstalkSources_1125941_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/c299e983-fefe-4d05-8e33-f8f318974e7d_isrForCrosstalkSources_1125941_16.stderr parsl.dataflow.dflow DEBUG: Task 2701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2701 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2701 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2701 has memoization hash 1a9a0b4b4cd357883d7303871843a397 parsl.dataflow.memoization INFO: Task 2701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2701 try 0 launched on executor multi with executor id 2702 parsl.dataflow.dflow INFO: Standard output for task 2701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/22d75827-d8b1-4b68-8e32-1ff2708c7b8c_isrForCrosstalkSources_1125892_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/22d75827-d8b1-4b68-8e32-1ff2708c7b8c_isrForCrosstalkSources_1125892_4.stderr parsl.dataflow.dflow DEBUG: Task 2702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2702 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2702 has memoization hash aa0498ff51ac8a2fc672168885f5c38d parsl.dataflow.memoization INFO: Task 2702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395705e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2702 try 0 launched on executor multi with executor id 2703 parsl.dataflow.dflow INFO: Standard output for task 2702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/23f9f7d5-0714-47ba-9da9-a904d790b8fb_isrForCrosstalkSources_1125906_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/23f9f7d5-0714-47ba-9da9-a904d790b8fb_isrForCrosstalkSources_1125906_47.stderr parsl.dataflow.dflow DEBUG: Task 2703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2703 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2703 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2703 has memoization hash df5502ed8e99e434f9ed3ee614c3c998 parsl.dataflow.memoization INFO: Task 2703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2703 try 0 launched on executor multi with executor id 2704 parsl.dataflow.dflow INFO: Standard output for task 2703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d18053ea-9adb-48f1-958e-58d90603cf1c_isrForCrosstalkSources_1125934_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d18053ea-9adb-48f1-958e-58d90603cf1c_isrForCrosstalkSources_1125934_16.stderr parsl.dataflow.dflow DEBUG: Task 2704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2704 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2704 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2704 has memoization hash 231d417a68fe8106e976d3d3705e008f parsl.dataflow.memoization INFO: Task 2704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395719e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2704 try 0 launched on executor multi with executor id 2705 parsl.dataflow.dflow INFO: Standard output for task 2704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/842b1e47-2ffa-442e-a19f-dccbaae95548_isrForCrosstalkSources_1125933_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/842b1e47-2ffa-442e-a19f-dccbaae95548_isrForCrosstalkSources_1125933_47.stderr parsl.dataflow.dflow DEBUG: Task 2705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2705 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2705 has memoization hash 2a7b304ce010b0b2ed23cd5dd4655b81 parsl.dataflow.memoization INFO: Task 2705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2705 try 0 launched on executor multi with executor id 2706 parsl.dataflow.dflow INFO: Standard output for task 2705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/7df485ed-834a-4f2e-9493-507088e09d4b_isrForCrosstalkSources_1125915_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/7df485ed-834a-4f2e-9493-507088e09d4b_isrForCrosstalkSources_1125915_61.stderr parsl.dataflow.dflow DEBUG: Task 2706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2706 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2706 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2706 has memoization hash 495074b2439689c077b51c7098143555 parsl.dataflow.memoization INFO: Task 2706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2706 try 0 launched on executor multi with executor id 2707 parsl.dataflow.dflow INFO: Standard output for task 2706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/f71b4797-a4dd-41b4-8b02-d28a8f7ce8fe_isrForCrosstalkSources_1125915_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/f71b4797-a4dd-41b4-8b02-d28a8f7ce8fe_isrForCrosstalkSources_1125915_51.stderr parsl.dataflow.dflow DEBUG: Task 2707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2707 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2707 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2707 has memoization hash f02389b1f5a91e5cb1493e82493bde96 parsl.dataflow.memoization INFO: Task 2707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2707 try 0 launched on executor multi with executor id 2708 parsl.dataflow.dflow INFO: Standard output for task 2707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/bd296f32-8423-4daf-a0c6-3a9b35ae0936_isrForCrosstalkSources_1125907_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/bd296f32-8423-4daf-a0c6-3a9b35ae0936_isrForCrosstalkSources_1125907_33.stderr parsl.dataflow.dflow DEBUG: Task 2708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2708 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2708 has memoization hash 8cc9b51a03d39a455ae983fb941ff689 parsl.dataflow.memoization INFO: Task 2708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2708 try 0 launched on executor multi with executor id 2709 parsl.dataflow.dflow INFO: Standard output for task 2708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/62922134-24e7-452e-b794-876194066d59_isrForCrosstalkSources_1125915_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/62922134-24e7-452e-b794-876194066d59_isrForCrosstalkSources_1125915_40.stderr parsl.dataflow.dflow DEBUG: Task 2709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2709 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2709 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2709 has memoization hash a82993e92a2e341f33353d80d3e35ff1 parsl.dataflow.memoization INFO: Task 2709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2709 try 0 launched on executor multi with executor id 2710 parsl.dataflow.dflow INFO: Standard output for task 2709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/df3d789d-4746-4ac9-b7e5-35878a20c064_isrForCrosstalkSources_1125912_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/df3d789d-4746-4ac9-b7e5-35878a20c064_isrForCrosstalkSources_1125912_30.stderr parsl.dataflow.dflow DEBUG: Task 2710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2710 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2710 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2710 has memoization hash 2b38e48fb3355537240584eb7dc64cd8 parsl.dataflow.memoization INFO: Task 2710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2710 try 0 launched on executor multi with executor id 2711 parsl.dataflow.dflow INFO: Standard output for task 2710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/417e8d4b-8176-424f-9304-3a662d63a45f_isrForCrosstalkSources_1125899_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/417e8d4b-8176-424f-9304-3a662d63a45f_isrForCrosstalkSources_1125899_60.stderr parsl.dataflow.dflow DEBUG: Task 2711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2711 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2711 has memoization hash 6b3cc94eb0589fcb2077048794215f66 parsl.dataflow.memoization INFO: Task 2711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2711 try 0 launched on executor multi with executor id 2712 parsl.dataflow.dflow INFO: Standard output for task 2711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e497055b-6510-4778-8db7-e27589fe7aef_isrForCrosstalkSources_1125916_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e497055b-6510-4778-8db7-e27589fe7aef_isrForCrosstalkSources_1125916_19.stderr parsl.dataflow.dflow DEBUG: Task 2712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2712 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2712 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2712 has memoization hash dd0bd10daf28550a01c03eb9090f1185 parsl.dataflow.memoization INFO: Task 2712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2712 try 0 launched on executor multi with executor id 2713 parsl.dataflow.dflow INFO: Standard output for task 2712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/71b578f8-e177-4c27-8c8e-697e9ce5b599_isrForCrosstalkSources_1125947_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/71b578f8-e177-4c27-8c8e-697e9ce5b599_isrForCrosstalkSources_1125947_55.stderr parsl.dataflow.dflow DEBUG: Task 2713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2713 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2713 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2713 has memoization hash c5c27dadff8ae4940f7e56d1b06cc78f parsl.dataflow.memoization INFO: Task 2713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2713 try 0 launched on executor multi with executor id 2714 parsl.dataflow.dflow INFO: Standard output for task 2713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/9d486282-6b5c-4be2-862e-075d85cc77b6_isrForCrosstalkSources_1125888_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/9d486282-6b5c-4be2-862e-075d85cc77b6_isrForCrosstalkSources_1125888_5.stderr parsl.dataflow.dflow DEBUG: Task 2714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2714 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2714 has memoization hash dfb14291e9ff7c87ff722b940ffeae57 parsl.dataflow.memoization INFO: Task 2714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2714 try 0 launched on executor multi with executor id 2715 parsl.dataflow.dflow INFO: Standard output for task 2714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/abfd5eae-ecf4-4ae6-9272-6f94c5779f94_isrForCrosstalkSources_1125908_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/abfd5eae-ecf4-4ae6-9272-6f94c5779f94_isrForCrosstalkSources_1125908_19.stderr parsl.dataflow.dflow DEBUG: Task 2715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2715 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2715 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2715 has memoization hash c92b1e8d949d10e39a2092119d0a9a8e parsl.dataflow.memoization INFO: Task 2715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2715 try 0 launched on executor multi with executor id 2716 parsl.dataflow.dflow INFO: Standard output for task 2715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/4819c1c7-9db2-4e23-820a-42234357c5c6_isrForCrosstalkSources_1125903_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/4819c1c7-9db2-4e23-820a-42234357c5c6_isrForCrosstalkSources_1125903_51.stderr parsl.dataflow.dflow DEBUG: Task 2716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2716 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2716 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2716 has memoization hash fb61cc9c84f091e903c405cf34e8de55 parsl.dataflow.memoization INFO: Task 2716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2716 try 0 launched on executor multi with executor id 2717 parsl.dataflow.dflow INFO: Standard output for task 2716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/dd0da4ee-0fcd-4968-b8ae-fe02989e6b7f_isrForCrosstalkSources_1125902_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/dd0da4ee-0fcd-4968-b8ae-fe02989e6b7f_isrForCrosstalkSources_1125902_33.stderr parsl.dataflow.dflow DEBUG: Task 2717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2717 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2717 has memoization hash 4d3ddc06bd4fd3ae582480b8bed1b856 parsl.dataflow.memoization INFO: Task 2717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2717 try 0 launched on executor multi with executor id 2718 parsl.dataflow.dflow INFO: Standard output for task 2717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/18e8226f-7ce4-421a-bbe1-6e162acf2621_isrForCrosstalkSources_1125905_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/18e8226f-7ce4-421a-bbe1-6e162acf2621_isrForCrosstalkSources_1125905_20.stderr parsl.dataflow.dflow DEBUG: Task 2718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2718 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2718 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2718 has memoization hash 8bbbfb7532c89e289d65b2327f3b2089 parsl.dataflow.memoization INFO: Task 2718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2718 try 0 launched on executor multi with executor id 2719 parsl.dataflow.dflow INFO: Standard output for task 2718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/6ccb60e1-1647-488b-983c-39780af3928b_isrForCrosstalkSources_1125921_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/6ccb60e1-1647-488b-983c-39780af3928b_isrForCrosstalkSources_1125921_32.stderr parsl.dataflow.dflow DEBUG: Task 2719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2719 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2719 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2719 has memoization hash 4898fa4beef36d572a5d20ebdbca5acf parsl.dataflow.memoization INFO: Task 2719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395727a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2719 try 0 launched on executor multi with executor id 2720 parsl.dataflow.dflow INFO: Standard output for task 2719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/b0c49bfe-8b59-476e-93ab-b90228dc9b64_isrForCrosstalkSources_1125898_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/b0c49bfe-8b59-476e-93ab-b90228dc9b64_isrForCrosstalkSources_1125898_43.stderr parsl.dataflow.dflow DEBUG: Task 2720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2720 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2720 has memoization hash cd619e119c32f021f5458305764e8e41 parsl.dataflow.memoization INFO: Task 2720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2720 try 0 launched on executor multi with executor id 2721 parsl.dataflow.dflow INFO: Standard output for task 2720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/1b0e4209-4d90-48be-95c7-aa469d69710b_isrForCrosstalkSources_1125928_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/1b0e4209-4d90-48be-95c7-aa469d69710b_isrForCrosstalkSources_1125928_38.stderr parsl.dataflow.dflow DEBUG: Task 2721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2721 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2721 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2721 has memoization hash 6b6f2a1ada4a9a33f3d1c958b9d1a6eb parsl.dataflow.memoization INFO: Task 2721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395728e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2721 try 0 launched on executor multi with executor id 2722 parsl.dataflow.dflow INFO: Standard output for task 2721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/64e3e9bb-28c4-429e-9eeb-feb0dd5c21f8_isrForCrosstalkSources_1125929_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/64e3e9bb-28c4-429e-9eeb-feb0dd5c21f8_isrForCrosstalkSources_1125929_43.stderr parsl.dataflow.dflow DEBUG: Task 2722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2722 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2722 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2722 has memoization hash 1f8b16f6a655b2444c37fb97a273668a parsl.dataflow.memoization INFO: Task 2722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2722 try 0 launched on executor multi with executor id 2723 parsl.dataflow.dflow INFO: Standard output for task 2722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/1d472ed3-0fed-434d-b57a-4ed0c79d7c95_isrForCrosstalkSources_1125902_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/1d472ed3-0fed-434d-b57a-4ed0c79d7c95_isrForCrosstalkSources_1125902_39.stderr parsl.dataflow.dflow DEBUG: Task 2723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2723 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2723 has memoization hash 98009721a7a0c25b3e16d10f49a18512 parsl.dataflow.memoization INFO: Task 2723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395725c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2723 try 0 launched on executor multi with executor id 2724 parsl.dataflow.dflow INFO: Standard output for task 2723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/dff3f5ce-4223-4a84-b0ae-7575f65722ac_isrForCrosstalkSources_1125906_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/dff3f5ce-4223-4a84-b0ae-7575f65722ac_isrForCrosstalkSources_1125906_1.stderr parsl.dataflow.dflow DEBUG: Task 2724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2724 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2724 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2724 has memoization hash 09f8df738923a9e459605d99dd8624d1 parsl.dataflow.memoization INFO: Task 2724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2724 try 0 launched on executor multi with executor id 2725 parsl.dataflow.dflow INFO: Standard output for task 2724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/796a8d10-f1b1-4e8e-9d21-7e7ba5fce27c_isrForCrosstalkSources_1125927_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/796a8d10-f1b1-4e8e-9d21-7e7ba5fce27c_isrForCrosstalkSources_1125927_10.stderr parsl.dataflow.dflow DEBUG: Task 2725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2725 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2725 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2725 has memoization hash 52a2306ac361c687cae5c2de3746de51 parsl.dataflow.memoization INFO: Task 2725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395723e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2725 try 0 launched on executor multi with executor id 2726 parsl.dataflow.dflow INFO: Standard output for task 2725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/23709c35-78ae-431b-bd85-df6ca8177e29_isrForCrosstalkSources_1125924_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/23709c35-78ae-431b-bd85-df6ca8177e29_isrForCrosstalkSources_1125924_28.stderr parsl.dataflow.dflow DEBUG: Task 2726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2726 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2726 has memoization hash a5af3aa25294585c1d4f7310dc13cbde parsl.dataflow.memoization INFO: Task 2726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2726 try 0 launched on executor multi with executor id 2727 parsl.dataflow.dflow INFO: Standard output for task 2726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/1a424be0-5f4e-47ca-80c3-7ff80d7a38b6_isrForCrosstalkSources_1125940_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/1a424be0-5f4e-47ca-80c3-7ff80d7a38b6_isrForCrosstalkSources_1125940_2.stderr parsl.dataflow.dflow DEBUG: Task 2727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2727 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2727 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2727 has memoization hash d228e699d7b8d4fbfb1ad3d09b123578 parsl.dataflow.memoization INFO: Task 2727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:00 proc_lsst.multi:170 [INFO] [multi] job_id 16896424 proc_lsst.multi INFO: [multi] job_id 16896424 2024-03-07 15:07:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:00 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896424 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896424 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 16896424 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-07 15:07:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-07 15:07:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2727 try 0 launched on executor multi with executor id 2728 parsl.dataflow.dflow INFO: Standard output for task 2727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/8a8805f5-b152-454c-bc2d-3a4d7cfc8c8e_isrForCrosstalkSources_1125946_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/8a8805f5-b152-454c-bc2d-3a4d7cfc8c8e_isrForCrosstalkSources_1125946_22.stderr parsl.dataflow.dflow DEBUG: Task 2728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2728 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2728 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2728 has memoization hash e03b560bf19d8c9b067941ea34a2d9f0 parsl.dataflow.memoization INFO: Task 2728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395722a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2728 try 0 launched on executor multi with executor id 2729 parsl.dataflow.dflow INFO: Standard output for task 2728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/d1e398b5-6307-41ed-96ba-29991475c775_isrForCrosstalkSources_1125918_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/d1e398b5-6307-41ed-96ba-29991475c775_isrForCrosstalkSources_1125918_38.stderr parsl.dataflow.dflow DEBUG: Task 2729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2729 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2729 has memoization hash ecff5967a11327ab2876826797d6a381 parsl.dataflow.memoization INFO: Task 2729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2729 try 0 launched on executor multi with executor id 2730 parsl.dataflow.dflow INFO: Standard output for task 2729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/f5f1ba00-35d5-441a-b48c-587118ce5607_isrForCrosstalkSources_1125946_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/f5f1ba00-35d5-441a-b48c-587118ce5607_isrForCrosstalkSources_1125946_40.stderr parsl.dataflow.dflow DEBUG: Task 2730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2730 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2730 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2730 has memoization hash 292d47197f4787794a931f7d75312cf8 parsl.dataflow.memoization INFO: Task 2730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395720c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2730 try 0 launched on executor multi with executor id 2731 parsl.dataflow.dflow INFO: Standard output for task 2730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/dc09485a-a1bd-4d3d-bb8e-c89442d0ecfc_isrForCrosstalkSources_1125931_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/dc09485a-a1bd-4d3d-bb8e-c89442d0ecfc_isrForCrosstalkSources_1125931_27.stderr parsl.dataflow.dflow DEBUG: Task 2731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2731 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2731 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2731 has memoization hash 489aaf59326904797cd253e393180055 parsl.dataflow.memoization INFO: Task 2731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395709a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2731 try 0 launched on executor multi with executor id 2732 parsl.dataflow.dflow INFO: Standard output for task 2731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/7d33f64e-8eb4-47df-afe5-24d19f0c9807_isrForCrosstalkSources_1125912_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/7d33f64e-8eb4-47df-afe5-24d19f0c9807_isrForCrosstalkSources_1125912_61.stderr parsl.dataflow.dflow DEBUG: Task 2732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2732 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2732 has memoization hash f850e11917ffdfd3a938367fde9f25c6 parsl.dataflow.memoization INFO: Task 2732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2732 try 0 launched on executor multi with executor id 2733 parsl.dataflow.dflow INFO: Standard output for task 2732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/863ebf22-c437-4a35-b035-c56f4c10a558_isrForCrosstalkSources_1125928_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/863ebf22-c437-4a35-b035-c56f4c10a558_isrForCrosstalkSources_1125928_60.stderr parsl.dataflow.dflow DEBUG: Task 2733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2733 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2733 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2733 has memoization hash 7ea698e5c0b5b4a00831c9472e63324e parsl.dataflow.memoization INFO: Task 2733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395707c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2733 try 0 launched on executor multi with executor id 2734 parsl.dataflow.dflow INFO: Standard output for task 2733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/43f8a1ab-10e2-41f5-b3a4-2f9e7b30876d_isrForCrosstalkSources_1125905_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/43f8a1ab-10e2-41f5-b3a4-2f9e7b30876d_isrForCrosstalkSources_1125905_36.stderr parsl.dataflow.dflow DEBUG: Task 2734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2734 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2734 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2734 has memoization hash c6679c979a12ffa03f9d7b2a6ccbc8a6 parsl.dataflow.memoization INFO: Task 2734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2734 try 0 launched on executor multi with executor id 2735 parsl.dataflow.dflow INFO: Standard output for task 2734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/4ba69243-cd00-4dfc-9148-1a6ba0930739_isrForCrosstalkSources_1125932_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/4ba69243-cd00-4dfc-9148-1a6ba0930739_isrForCrosstalkSources_1125932_46.stderr parsl.dataflow.dflow DEBUG: Task 2735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2735 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2735 has memoization hash 83c892339e9cbbe48a3717bca021be0d parsl.dataflow.memoization INFO: Task 2735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2735 try 0 launched on executor multi with executor id 2736 parsl.dataflow.dflow INFO: Standard output for task 2735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/abcc6bab-ad4b-4a38-a166-3011f92d2543_isrForCrosstalkSources_1125946_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/abcc6bab-ad4b-4a38-a166-3011f92d2543_isrForCrosstalkSources_1125946_20.stderr parsl.dataflow.dflow DEBUG: Task 2736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2736 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2736 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2736 has memoization hash 47184abb9aa5fac1e5dfc346e53900d7 parsl.dataflow.memoization INFO: Task 2736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39572020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2736 try 0 launched on executor multi with executor id 2737 parsl.dataflow.dflow INFO: Standard output for task 2736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/10d84562-ab83-434e-aa0c-3ca7a9f39cc0_isrForCrosstalkSources_1125937_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/10d84562-ab83-434e-aa0c-3ca7a9f39cc0_isrForCrosstalkSources_1125937_22.stderr parsl.dataflow.dflow DEBUG: Task 2737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2737 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2737 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2737 has memoization hash 40979990b2f483db7e80dda946b8e645 parsl.dataflow.memoization INFO: Task 2737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2737 try 0 launched on executor multi with executor id 2738 parsl.dataflow.dflow INFO: Standard output for task 2737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/889f91c1-39b8-433c-b62e-8136e76bbd37_isrForCrosstalkSources_1125888_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/889f91c1-39b8-433c-b62e-8136e76bbd37_isrForCrosstalkSources_1125888_58.stderr parsl.dataflow.dflow DEBUG: Task 2738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2738 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2738 has memoization hash 0bd5dce3dd2492d589829ae132db6b06 parsl.dataflow.memoization INFO: Task 2738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2738 try 0 launched on executor multi with executor id 2739 parsl.dataflow.dflow INFO: Standard output for task 2738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/97232421-cab8-4125-93bd-277c158c4dee_isrForCrosstalkSources_1125897_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/97232421-cab8-4125-93bd-277c158c4dee_isrForCrosstalkSources_1125897_9.stderr parsl.dataflow.dflow DEBUG: Task 2739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2739 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2739 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2739 has memoization hash be99958532dcb787d1f9f0f9f9f48b9c parsl.dataflow.memoization INFO: Task 2739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2739 try 0 launched on executor multi with executor id 2740 parsl.dataflow.dflow INFO: Standard output for task 2739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/1829332f-a6bc-422a-ae6d-15119a4185e8_isrForCrosstalkSources_1125899_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/1829332f-a6bc-422a-ae6d-15119a4185e8_isrForCrosstalkSources_1125899_56.stderr parsl.dataflow.dflow DEBUG: Task 2740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2740 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2740 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2740 has memoization hash 7ee4f9fc445468dc38c32ce01b261b00 parsl.dataflow.memoization INFO: Task 2740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39571d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2740 try 0 launched on executor multi with executor id 2741 parsl.dataflow.dflow INFO: Standard output for task 2740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/124388d2-e7f6-4e73-a3a0-513f4198e3a6_isrForCrosstalkSources_1125908_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/124388d2-e7f6-4e73-a3a0-513f4198e3a6_isrForCrosstalkSources_1125908_48.stderr parsl.dataflow.dflow DEBUG: Task 2741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2741 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2741 has memoization hash f620c8f7d738fc410695603a5364c7fb parsl.dataflow.memoization INFO: Task 2741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2741 try 0 launched on executor multi with executor id 2742 parsl.dataflow.dflow INFO: Standard output for task 2741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/0b8fb144-2047-4dd2-baaa-b89dc6ec28b6_isrForCrosstalkSources_1125901_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/0b8fb144-2047-4dd2-baaa-b89dc6ec28b6_isrForCrosstalkSources_1125901_55.stderr parsl.dataflow.dflow DEBUG: Task 2742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2742 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2742 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2742 has memoization hash ba6cc92b2f06798f17951a85dabcdbc8 parsl.dataflow.memoization INFO: Task 2742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2742 try 0 launched on executor multi with executor id 2743 parsl.dataflow.dflow INFO: Standard output for task 2742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/1bf96037-d0a0-40a3-9ac9-03730960522b_isrForCrosstalkSources_1125947_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/1bf96037-d0a0-40a3-9ac9-03730960522b_isrForCrosstalkSources_1125947_41.stderr parsl.dataflow.dflow DEBUG: Task 2743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2743 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2743 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2743 has memoization hash 58aa33463ac2fed56074eefd26db9a78 parsl.dataflow.memoization INFO: Task 2743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2743 try 0 launched on executor multi with executor id 2744 parsl.dataflow.dflow INFO: Standard output for task 2743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/ba5d58db-4484-4cb4-9ad2-988302f7d181_isrForCrosstalkSources_1125929_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/ba5d58db-4484-4cb4-9ad2-988302f7d181_isrForCrosstalkSources_1125929_57.stderr parsl.dataflow.dflow DEBUG: Task 2744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2744 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2744 has memoization hash 07549bb201590e35a3d7c1fcd917cd61 parsl.dataflow.memoization INFO: Task 2744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2744 try 0 launched on executor multi with executor id 2745 parsl.dataflow.dflow INFO: Standard output for task 2744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/72a38c02-b489-4bec-920a-8fbacf0c57d7_isrForCrosstalkSources_1125902_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/72a38c02-b489-4bec-920a-8fbacf0c57d7_isrForCrosstalkSources_1125902_4.stderr parsl.dataflow.dflow DEBUG: Task 2745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2745 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2745 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2745 has memoization hash 33fb57eded4ddcebdb3fa906f069746f parsl.dataflow.memoization INFO: Task 2745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395736a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2745 try 0 launched on executor multi with executor id 2746 parsl.dataflow.dflow INFO: Standard output for task 2745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/c0607eb7-0267-4a80-a0a3-4b1699278326_isrForCrosstalkSources_1125898_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/c0607eb7-0267-4a80-a0a3-4b1699278326_isrForCrosstalkSources_1125898_31.stderr parsl.dataflow.dflow DEBUG: Task 2746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2746 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2746 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2746 has memoization hash d6872c2238321e463a9ec4cb796a1a7a parsl.dataflow.memoization INFO: Task 2746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2746 try 0 launched on executor multi with executor id 2747 parsl.dataflow.dflow INFO: Standard output for task 2746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/7a0be123-b460-4f9c-a37c-dbda44f93a5e_isrForCrosstalkSources_1125933_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/7a0be123-b460-4f9c-a37c-dbda44f93a5e_isrForCrosstalkSources_1125933_1.stderr parsl.dataflow.dflow DEBUG: Task 2747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2747 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2747 has memoization hash 9abc0249b7e84b88068063a7bdfc5294 parsl.dataflow.memoization INFO: Task 2747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395734c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2747 try 0 launched on executor multi with executor id 2748 parsl.dataflow.dflow INFO: Standard output for task 2747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ccc1a43f-4bf6-4dc0-9002-92fd5f108931_isrForCrosstalkSources_1125939_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ccc1a43f-4bf6-4dc0-9002-92fd5f108931_isrForCrosstalkSources_1125939_53.stderr parsl.dataflow.dflow DEBUG: Task 2748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2748 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2748 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2748 has memoization hash 8cb726e6f720901d86178fa22cae4686 parsl.dataflow.memoization INFO: Task 2748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395737e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2748 try 0 launched on executor multi with executor id 2749 parsl.dataflow.dflow INFO: Standard output for task 2748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/bbd3a262-90ae-4eab-8b3f-5be3f0a6a1de_isrForCrosstalkSources_1125917_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/bbd3a262-90ae-4eab-8b3f-5be3f0a6a1de_isrForCrosstalkSources_1125917_61.stderr parsl.dataflow.dflow DEBUG: Task 2749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2749 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2749 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2749 has memoization hash 111dd858d56bb005a215cf4876a2a828 parsl.dataflow.memoization INFO: Task 2749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2749 try 0 launched on executor multi with executor id 2750 parsl.dataflow.dflow INFO: Standard output for task 2749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/4749fec9-29e1-4e8e-863b-e9be2bd9c2e4_isrForCrosstalkSources_1125910_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/4749fec9-29e1-4e8e-863b-e9be2bd9c2e4_isrForCrosstalkSources_1125910_59.stderr parsl.dataflow.dflow DEBUG: Task 2750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2750 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2750 has memoization hash 4e44055ce9933214b706fe3e6bbed3e1 parsl.dataflow.memoization INFO: Task 2750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2750 try 0 launched on executor multi with executor id 2751 parsl.dataflow.dflow INFO: Standard output for task 2750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/2503599a-0c6b-432e-9f53-be9552cefdf1_isrForCrosstalkSources_1125922_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/2503599a-0c6b-432e-9f53-be9552cefdf1_isrForCrosstalkSources_1125922_55.stderr parsl.dataflow.dflow DEBUG: Task 2751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2751 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2751 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2751 has memoization hash 3a5e406d2ba111c0d93b1174726d2bb3 parsl.dataflow.memoization INFO: Task 2751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2751 try 0 launched on executor multi with executor id 2752 parsl.dataflow.dflow INFO: Standard output for task 2751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/38ff2949-383c-400a-816f-4b842240f710_isrForCrosstalkSources_1125903_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/38ff2949-383c-400a-816f-4b842240f710_isrForCrosstalkSources_1125903_10.stderr parsl.dataflow.dflow DEBUG: Task 2752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2752 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2752 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2752 has memoization hash 21afee80b277fe1b17eaf0a11cbad5f2 parsl.dataflow.memoization INFO: Task 2752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2752 try 0 launched on executor multi with executor id 2753 parsl.dataflow.dflow INFO: Standard output for task 2752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/6f4a180d-6208-48b5-aa1c-b4ec22a6d1ab_isrForCrosstalkSources_1125942_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/6f4a180d-6208-48b5-aa1c-b4ec22a6d1ab_isrForCrosstalkSources_1125942_42.stderr parsl.dataflow.dflow DEBUG: Task 2753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2753 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2753 has memoization hash 0ba009d64c70bdab936d9c53b233d272 parsl.dataflow.memoization INFO: Task 2753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2753 try 0 launched on executor multi with executor id 2754 parsl.dataflow.dflow INFO: Standard output for task 2753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/977fcf50-ef21-46f1-b865-047cfd7b3fc2_isrForCrosstalkSources_1125933_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/977fcf50-ef21-46f1-b865-047cfd7b3fc2_isrForCrosstalkSources_1125933_35.stderr parsl.dataflow.dflow DEBUG: Task 2754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2754 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2754 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2754 has memoization hash d49e6c5b937e4e517e473e7a9c9a1efa parsl.dataflow.memoization INFO: Task 2754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2754 try 0 launched on executor multi with executor id 2755 parsl.dataflow.dflow INFO: Standard output for task 2754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/25ea66ea-835b-4fdf-9980-dcb89545f870_isrForCrosstalkSources_1125910_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/25ea66ea-835b-4fdf-9980-dcb89545f870_isrForCrosstalkSources_1125910_22.stderr parsl.dataflow.dflow DEBUG: Task 2755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2755 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2755 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2755 has memoization hash d9c1da9db8078d099c03fb17a6879388 parsl.dataflow.memoization INFO: Task 2755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2755 try 0 launched on executor multi with executor id 2756 parsl.dataflow.dflow INFO: Standard output for task 2755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/ff39a3c1-b884-4148-9426-d4079959ecf4_isrForCrosstalkSources_1125943_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/ff39a3c1-b884-4148-9426-d4079959ecf4_isrForCrosstalkSources_1125943_46.stderr parsl.dataflow.dflow DEBUG: Task 2756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2756 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2756 has memoization hash 007e43f28e5514a0a3dffe42a9523449 parsl.dataflow.memoization INFO: Task 2756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2756 try 0 launched on executor multi with executor id 2757 parsl.dataflow.dflow INFO: Standard output for task 2756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/4638d7f3-3ca2-4776-998b-9d2d79305bb8_isrForCrosstalkSources_1125946_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/4638d7f3-3ca2-4776-998b-9d2d79305bb8_isrForCrosstalkSources_1125946_50.stderr parsl.dataflow.dflow DEBUG: Task 2757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2757 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2757 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2757 has memoization hash dfd551377e4c1d150fc215a1be569389 parsl.dataflow.memoization INFO: Task 2757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395739c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2757 try 0 launched on executor multi with executor id 2758 parsl.dataflow.dflow INFO: Standard output for task 2757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/cceec988-e459-428b-b223-a3b9d9d782ef_isrForCrosstalkSources_1125921_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/cceec988-e459-428b-b223-a3b9d9d782ef_isrForCrosstalkSources_1125921_14.stderr parsl.dataflow.dflow DEBUG: Task 2758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2758 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2758 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2758 has memoization hash 5d23bb0dc71440d4af72dfd3da713eab parsl.dataflow.memoization INFO: Task 2758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2758 try 0 launched on executor multi with executor id 2759 parsl.dataflow.dflow INFO: Standard output for task 2758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/2bb3120f-ebc8-49c9-8da7-f5232a14a92a_isrForCrosstalkSources_1125897_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/2bb3120f-ebc8-49c9-8da7-f5232a14a92a_isrForCrosstalkSources_1125897_2.stderr parsl.dataflow.dflow DEBUG: Task 2759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2759 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2759 has memoization hash b241bfd41a62d8f38013a23293cd26c3 parsl.dataflow.memoization INFO: Task 2759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39570c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2759 try 0 launched on executor multi with executor id 2760 parsl.dataflow.dflow INFO: Standard output for task 2759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/7386a5d1-2006-453c-99df-4b870d830d81_isrForCrosstalkSources_1125930_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/7386a5d1-2006-453c-99df-4b870d830d81_isrForCrosstalkSources_1125930_7.stderr parsl.dataflow.dflow DEBUG: Task 2760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2760 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2760 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2760 has memoization hash 9259b0f6e61ec8a125652210ac417c19 parsl.dataflow.memoization INFO: Task 2760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2760 try 0 launched on executor multi with executor id 2761 parsl.dataflow.dflow INFO: Standard output for task 2760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/b996cee8-214d-412c-b52b-08b6f2504df1_isrForCrosstalkSources_1125902_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/b996cee8-214d-412c-b52b-08b6f2504df1_isrForCrosstalkSources_1125902_61.stderr parsl.dataflow.dflow DEBUG: Task 2761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2761 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2761 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2761 has memoization hash 138e09a7dcbc457932aef4b7e56ab6a6 parsl.dataflow.memoization INFO: Task 2761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2761 try 0 launched on executor multi with executor id 2762 parsl.dataflow.dflow INFO: Standard output for task 2761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/4ffa531b-b305-4d70-9edd-cb608dbeabf3_isrForCrosstalkSources_1125909_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/4ffa531b-b305-4d70-9edd-cb608dbeabf3_isrForCrosstalkSources_1125909_23.stderr parsl.dataflow.dflow DEBUG: Task 2762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2762 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2762 has memoization hash fc74bc9392fd5317ef57a9dc31d7c9ab parsl.dataflow.memoization INFO: Task 2762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2762 try 0 launched on executor multi with executor id 2763 parsl.dataflow.dflow INFO: Standard output for task 2762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/1a86f032-9b06-4fa5-a4c8-63225f209a5d_isrForCrosstalkSources_1125944_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/1a86f032-9b06-4fa5-a4c8-63225f209a5d_isrForCrosstalkSources_1125944_27.stderr parsl.dataflow.dflow DEBUG: Task 2763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2763 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2763 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2763 has memoization hash ae0567856108e11eaa4b3568bc6dc96e parsl.dataflow.memoization INFO: Task 2763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2763 try 0 launched on executor multi with executor id 2764 parsl.dataflow.dflow INFO: Standard output for task 2763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/e775a41a-3b4f-449c-b0e2-12fba4d8b272_isrForCrosstalkSources_1125898_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/e775a41a-3b4f-449c-b0e2-12fba4d8b272_isrForCrosstalkSources_1125898_38.stderr parsl.dataflow.dflow DEBUG: Task 2764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2764 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2764 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2764 has memoization hash cd1b3310057a34187bde5c14e130ed7b parsl.dataflow.memoization INFO: Task 2764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2764 try 0 launched on executor multi with executor id 2765 parsl.dataflow.dflow INFO: Standard output for task 2764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/b69a0e1b-bfa6-40d6-a484-820457e030b0_isrForCrosstalkSources_1125907_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/b69a0e1b-bfa6-40d6-a484-820457e030b0_isrForCrosstalkSources_1125907_21.stderr parsl.dataflow.dflow DEBUG: Task 2765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2765 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2765 has memoization hash e2135435e031f14f19fd387706febb53 parsl.dataflow.memoization INFO: Task 2765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2765 try 0 launched on executor multi with executor id 2766 parsl.dataflow.dflow INFO: Standard output for task 2765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/d21e0028-ed08-437e-8658-af5bbfa01b68_isrForCrosstalkSources_1125901_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/d21e0028-ed08-437e-8658-af5bbfa01b68_isrForCrosstalkSources_1125901_35.stderr parsl.dataflow.dflow DEBUG: Task 2766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2766 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2766 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2766 has memoization hash f959375bfb8fbd2674cb37260525ae2d parsl.dataflow.memoization INFO: Task 2766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2766 try 0 launched on executor multi with executor id 2767 parsl.dataflow.dflow INFO: Standard output for task 2766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/39aa9f0b-8f4c-4b86-b118-41cf10d0fb51_isrForCrosstalkSources_1125924_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/39aa9f0b-8f4c-4b86-b118-41cf10d0fb51_isrForCrosstalkSources_1125924_44.stderr parsl.dataflow.dflow DEBUG: Task 2767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2767 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2767 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2767 has memoization hash 528c04c5f99670d783a7c24bb81816ab parsl.dataflow.memoization INFO: Task 2767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2767 try 0 launched on executor multi with executor id 2768 parsl.dataflow.dflow INFO: Standard output for task 2767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/7f09cd96-63fd-4871-ad2b-d2dfa672d3e9_isrForCrosstalkSources_1125904_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/7f09cd96-63fd-4871-ad2b-d2dfa672d3e9_isrForCrosstalkSources_1125904_23.stderr parsl.dataflow.dflow DEBUG: Task 2768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2768 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2768 has memoization hash 0e497cd8cc6fcaf473b5b78ffa3c0576 parsl.dataflow.memoization INFO: Task 2768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2768 try 0 launched on executor multi with executor id 2769 parsl.dataflow.dflow INFO: Standard output for task 2768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2d51edac-ae6c-47ad-b0f5-92d4eaf99a24_isrForCrosstalkSources_1125933_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2d51edac-ae6c-47ad-b0f5-92d4eaf99a24_isrForCrosstalkSources_1125933_28.stderr parsl.dataflow.dflow DEBUG: Task 2769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2769 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2769 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2769 has memoization hash 450f30285af72286be22f0dfc3235612 parsl.dataflow.memoization INFO: Task 2769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2769 try 0 launched on executor multi with executor id 2770 parsl.dataflow.dflow INFO: Standard output for task 2769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/91398b70-7e05-41e1-8709-8fef969e286f_isrForCrosstalkSources_1125936_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/91398b70-7e05-41e1-8709-8fef969e286f_isrForCrosstalkSources_1125936_22.stderr parsl.dataflow.dflow DEBUG: Task 2770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2770 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2770 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2770 has memoization hash 5b8ea6024a8b6503fb950ce3ab70a096 parsl.dataflow.memoization INFO: Task 2770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2770 try 0 launched on executor multi with executor id 2771 parsl.dataflow.dflow INFO: Standard output for task 2770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/e43ca15d-4859-42da-b85e-10c4fb8ca093_isrForCrosstalkSources_1125888_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/e43ca15d-4859-42da-b85e-10c4fb8ca093_isrForCrosstalkSources_1125888_32.stderr parsl.dataflow.dflow DEBUG: Task 2771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2771 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2771 has memoization hash 4555a7b1327bdd72688fe58327c74721 parsl.dataflow.memoization INFO: Task 2771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2771 try 0 launched on executor multi with executor id 2772 parsl.dataflow.dflow INFO: Standard output for task 2771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/2b1d8841-0ba5-4c02-ae07-e0a5fd08d78d_isrForCrosstalkSources_1125932_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/2b1d8841-0ba5-4c02-ae07-e0a5fd08d78d_isrForCrosstalkSources_1125932_56.stderr parsl.dataflow.dflow DEBUG: Task 2772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2772 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2772 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2772 has memoization hash 339bbba20f6f3cb1625cbc55051a0c5e parsl.dataflow.memoization INFO: Task 2772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2772 try 0 launched on executor multi with executor id 2773 parsl.dataflow.dflow INFO: Standard output for task 2772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/5142e845-e6c7-421d-a8d3-46d0e516c6ad_isrForCrosstalkSources_1125903_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/5142e845-e6c7-421d-a8d3-46d0e516c6ad_isrForCrosstalkSources_1125903_8.stderr parsl.dataflow.dflow DEBUG: Task 2773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2773 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2773 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2773 has memoization hash 111a24fd456912818ca176d395ad8004 parsl.dataflow.memoization INFO: Task 2773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2773 try 0 launched on executor multi with executor id 2774 parsl.dataflow.dflow INFO: Standard output for task 2773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/9b8f1ddd-c493-4da5-b351-32c92f4f0a60_isrForCrosstalkSources_1125927_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/9b8f1ddd-c493-4da5-b351-32c92f4f0a60_isrForCrosstalkSources_1125927_31.stderr parsl.dataflow.dflow DEBUG: Task 2774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2774 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2774 has memoization hash 343bb129276a97f10106c12920c05be6 parsl.dataflow.memoization INFO: Task 2774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:01 proc_lsst.multi:170 [INFO] [multi] job_id 16896425 proc_lsst.multi INFO: [multi] job_id 16896425 2024-03-07 15:07:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 15:07:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:01 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896425 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896425 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 16896425 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-07 15:07:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-07 15:07:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 15:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2774 try 0 launched on executor multi with executor id 2775 parsl.dataflow.dflow INFO: Standard output for task 2774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/df3003e6-fc0d-4d72-8c7f-97c3c93a8be9_isrForCrosstalkSources_1125914_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/df3003e6-fc0d-4d72-8c7f-97c3c93a8be9_isrForCrosstalkSources_1125914_3.stderr parsl.dataflow.dflow DEBUG: Task 2775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2775 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2775 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2775 has memoization hash ea80c34d7da0e57830acff68630a9602 parsl.dataflow.memoization INFO: Task 2775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a395732e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2775 try 0 launched on executor multi with executor id 2776 parsl.dataflow.dflow INFO: Standard output for task 2775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/45b0ba7e-b9e8-432a-8667-e46b2e859e55_isrForCrosstalkSources_1125923_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/45b0ba7e-b9e8-432a-8667-e46b2e859e55_isrForCrosstalkSources_1125923_53.stderr parsl.dataflow.dflow DEBUG: Task 2776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2776 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2776 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2776 has memoization hash e7a4cd55949585b4518f1e5506340221 parsl.dataflow.memoization INFO: Task 2776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2776 try 0 launched on executor multi with executor id 2777 parsl.dataflow.dflow INFO: Standard output for task 2776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/174f8687-8032-4102-ad15-61f4a71cafa1_isrForCrosstalkSources_1125947_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/174f8687-8032-4102-ad15-61f4a71cafa1_isrForCrosstalkSources_1125947_36.stderr parsl.dataflow.dflow DEBUG: Task 2777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2777 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2777 has memoization hash 173c0bfaed0e417f79fb5078bc54ffea parsl.dataflow.memoization INFO: Task 2777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39573380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2777 try 0 launched on executor multi with executor id 2778 parsl.dataflow.dflow INFO: Standard output for task 2777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/27824355-e134-450b-b0ea-9bd944876f56_isrForCrosstalkSources_1125907_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/27824355-e134-450b-b0ea-9bd944876f56_isrForCrosstalkSources_1125907_61.stderr parsl.dataflow.dflow DEBUG: Task 2778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2778 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2778 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2778 has memoization hash 05db3533168588921dbfa68201533c86 parsl.dataflow.memoization INFO: Task 2778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2778 try 0 launched on executor multi with executor id 2779 parsl.dataflow.dflow INFO: Standard output for task 2778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/f6f8fc68-65a4-4920-a68f-17c069f3ce7c_isrForCrosstalkSources_1125920_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/f6f8fc68-65a4-4920-a68f-17c069f3ce7c_isrForCrosstalkSources_1125920_14.stderr parsl.dataflow.dflow DEBUG: Task 2779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2779 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2779 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2779 has memoization hash 379246d98f10bee71aa0d866aeb79513 parsl.dataflow.memoization INFO: Task 2779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2779 try 0 launched on executor multi with executor id 2780 parsl.dataflow.dflow INFO: Standard output for task 2779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/ca4d024e-2c8d-4b36-94cd-cdb21cc15dd7_isrForCrosstalkSources_1125926_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/ca4d024e-2c8d-4b36-94cd-cdb21cc15dd7_isrForCrosstalkSources_1125926_56.stderr parsl.dataflow.dflow DEBUG: Task 2780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2780 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2780 has memoization hash 3fe344bfcd978d73a592a670c13b8721 parsl.dataflow.memoization INFO: Task 2780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2780 try 0 launched on executor multi with executor id 2781 parsl.dataflow.dflow INFO: Standard output for task 2780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/aeaf92ec-cf53-45af-83f4-25607fd74a15_isrForCrosstalkSources_1125897_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/aeaf92ec-cf53-45af-83f4-25607fd74a15_isrForCrosstalkSources_1125897_1.stderr parsl.dataflow.dflow DEBUG: Task 2781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2781 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2781 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2781 has memoization hash 6add338e3f8d3ec2ff0582f77734eb3c parsl.dataflow.memoization INFO: Task 2781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2781 try 0 launched on executor multi with executor id 2782 parsl.dataflow.dflow INFO: Standard output for task 2781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/533af3e2-a175-456f-b727-4764d0f1cc30_isrForCrosstalkSources_1125920_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/533af3e2-a175-456f-b727-4764d0f1cc30_isrForCrosstalkSources_1125920_52.stderr parsl.dataflow.dflow DEBUG: Task 2782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2782 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2782 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2782 has memoization hash c79721768862789ebabeab1830392159 parsl.dataflow.memoization INFO: Task 2782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2782 try 0 launched on executor multi with executor id 2783 parsl.dataflow.dflow INFO: Standard output for task 2782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/762a2b15-3311-4fa7-848b-daca12129a6a_isrForCrosstalkSources_1125892_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/762a2b15-3311-4fa7-848b-daca12129a6a_isrForCrosstalkSources_1125892_19.stderr parsl.dataflow.dflow DEBUG: Task 2783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2783 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2783 has memoization hash 6607bc192a44a345af9cb1603cbd419f parsl.dataflow.memoization INFO: Task 2783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2783 try 0 launched on executor multi with executor id 2784 parsl.dataflow.dflow INFO: Standard output for task 2783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/7c948243-ff04-491b-af04-82505e444ed3_isrForCrosstalkSources_1125907_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/7c948243-ff04-491b-af04-82505e444ed3_isrForCrosstalkSources_1125907_38.stderr parsl.dataflow.dflow DEBUG: Task 2784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2784 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2784 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2784 has memoization hash 5767b9d70637db88f8c1d94e2019603e parsl.dataflow.memoization INFO: Task 2784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2784 try 0 launched on executor multi with executor id 2785 parsl.dataflow.dflow INFO: Standard output for task 2784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/319c4c7e-d46e-400a-aebc-a31c2382b76e_isrForCrosstalkSources_1125920_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/319c4c7e-d46e-400a-aebc-a31c2382b76e_isrForCrosstalkSources_1125920_50.stderr parsl.dataflow.dflow DEBUG: Task 2785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2785 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2785 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2785 has memoization hash 5c0b20c85405ffe207d71ba070bd4475 parsl.dataflow.memoization INFO: Task 2785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2785 try 0 launched on executor multi with executor id 2786 parsl.dataflow.dflow INFO: Standard output for task 2785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a2895cc3-aec8-4874-88f1-b77712e97405_isrForCrosstalkSources_1125900_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a2895cc3-aec8-4874-88f1-b77712e97405_isrForCrosstalkSources_1125900_48.stderr parsl.dataflow.dflow DEBUG: Task 2786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2786 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2786 has memoization hash 0bc09cc2d0addb7dfdb5175d864cab72 parsl.dataflow.memoization INFO: Task 2786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2786 try 0 launched on executor multi with executor id 2787 parsl.dataflow.dflow INFO: Standard output for task 2786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/99c12d27-f111-4f54-9840-da29960abd08_isrForCrosstalkSources_1125906_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/99c12d27-f111-4f54-9840-da29960abd08_isrForCrosstalkSources_1125906_54.stderr parsl.dataflow.dflow DEBUG: Task 2787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2787 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2787 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2787 has memoization hash d2c8b3aff2f903f8afe0065b254ddc2f parsl.dataflow.memoization INFO: Task 2787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2787 try 0 launched on executor multi with executor id 2788 parsl.dataflow.dflow INFO: Standard output for task 2787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/f048a2f6-9270-42e2-b257-66af6e012086_isrForCrosstalkSources_1125891_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/f048a2f6-9270-42e2-b257-66af6e012086_isrForCrosstalkSources_1125891_61.stderr parsl.dataflow.dflow DEBUG: Task 2788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2788 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2788 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2788 has memoization hash df1febd32ac887260085cf548319a759 parsl.dataflow.memoization INFO: Task 2788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2788 try 0 launched on executor multi with executor id 2789 parsl.dataflow.dflow INFO: Standard output for task 2788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/40c31a75-8013-4319-9051-e67b0058fefb_isrForCrosstalkSources_1125903_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/40c31a75-8013-4319-9051-e67b0058fefb_isrForCrosstalkSources_1125903_39.stderr parsl.dataflow.dflow DEBUG: Task 2789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2789 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2789 has memoization hash a78723634ebeaf48b8af1f55dd23b4fe parsl.dataflow.memoization INFO: Task 2789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2789 try 0 launched on executor multi with executor id 2790 parsl.dataflow.dflow INFO: Standard output for task 2789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/0534139e-6f1f-4f51-a956-32a34b0e1d4b_isrForCrosstalkSources_1125944_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/0534139e-6f1f-4f51-a956-32a34b0e1d4b_isrForCrosstalkSources_1125944_18.stderr parsl.dataflow.dflow DEBUG: Task 2790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2790 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2790 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2790 has memoization hash 2071a072e1c6435fbb9ff5d9930705a3 parsl.dataflow.memoization INFO: Task 2790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2790 try 0 launched on executor multi with executor id 2791 parsl.dataflow.dflow INFO: Standard output for task 2790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/1a218824-5745-4d75-ae14-c29312500c29_isrForCrosstalkSources_1125935_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/1a218824-5745-4d75-ae14-c29312500c29_isrForCrosstalkSources_1125935_10.stderr parsl.dataflow.dflow DEBUG: Task 2791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2791 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2791 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2791 has memoization hash 8888d3f7f64386b7c7acb0672af8d3c6 parsl.dataflow.memoization INFO: Task 2791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2791 try 0 launched on executor multi with executor id 2792 parsl.dataflow.dflow INFO: Standard output for task 2791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/b740d282-90d4-4a82-90bd-c38583c0439f_isrForCrosstalkSources_1125898_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/b740d282-90d4-4a82-90bd-c38583c0439f_isrForCrosstalkSources_1125898_10.stderr parsl.dataflow.dflow DEBUG: Task 2792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2792 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2792 has memoization hash db6d68d6b5b330c58f8abf8b5a929f34 parsl.dataflow.memoization INFO: Task 2792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2792 try 0 launched on executor multi with executor id 2793 parsl.dataflow.dflow INFO: Standard output for task 2792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/c6d58860-543d-4ea4-bd17-a6fb697d2290_isrForCrosstalkSources_1125939_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/c6d58860-543d-4ea4-bd17-a6fb697d2290_isrForCrosstalkSources_1125939_18.stderr parsl.dataflow.dflow DEBUG: Task 2793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2793 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2793 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2793 has memoization hash f07c10c64e986a2b15a0754e775f2e16 parsl.dataflow.memoization INFO: Task 2793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2793 try 0 launched on executor multi with executor id 2794 parsl.dataflow.dflow INFO: Standard output for task 2793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/e10c35c3-53af-4cb8-973d-ff5b9dd871eb_isrForCrosstalkSources_1125946_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/e10c35c3-53af-4cb8-973d-ff5b9dd871eb_isrForCrosstalkSources_1125946_41.stderr parsl.dataflow.dflow DEBUG: Task 2794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2794 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2794 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2794 has memoization hash a8d47b729e6560e7f36ab7dacb2678ea parsl.dataflow.memoization INFO: Task 2794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2794 try 0 launched on executor multi with executor id 2795 parsl.dataflow.dflow INFO: Standard output for task 2794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/a0ee050e-b132-4c67-a6d8-b6c2f6b07482_isrForCrosstalkSources_1125936_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/a0ee050e-b132-4c67-a6d8-b6c2f6b07482_isrForCrosstalkSources_1125936_6.stderr parsl.dataflow.dflow DEBUG: Task 2795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2795 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2795 has memoization hash 1a6b9fba806c5aa47159b8c011e5ae2a parsl.dataflow.memoization INFO: Task 2795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2795 try 0 launched on executor multi with executor id 2796 parsl.dataflow.dflow INFO: Standard output for task 2795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/5f4dbea9-5458-4ed8-bb67-e18f66c0610d_isrForCrosstalkSources_1125924_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/5f4dbea9-5458-4ed8-bb67-e18f66c0610d_isrForCrosstalkSources_1125924_1.stderr parsl.dataflow.dflow DEBUG: Task 2796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2796 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2796 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2796 has memoization hash 713322984a9b1a61e24f41ed44fc17a2 parsl.dataflow.memoization INFO: Task 2796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2796 try 0 launched on executor multi with executor id 2797 parsl.dataflow.dflow INFO: Standard output for task 2796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/546543ac-e95c-43b5-8c63-83e83edbfa2b_isrForCrosstalkSources_1125935_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/546543ac-e95c-43b5-8c63-83e83edbfa2b_isrForCrosstalkSources_1125935_13.stderr parsl.dataflow.dflow DEBUG: Task 2797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2797 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2797 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2797 has memoization hash 0cb2e9e4b842b687a12e2113af508f6e parsl.dataflow.memoization INFO: Task 2797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2797 try 0 launched on executor multi with executor id 2798 parsl.dataflow.dflow INFO: Standard output for task 2797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/4db6714c-7c98-4d37-ade7-2e169c680e1d_isrForCrosstalkSources_1125917_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/4db6714c-7c98-4d37-ade7-2e169c680e1d_isrForCrosstalkSources_1125917_57.stderr parsl.dataflow.dflow DEBUG: Task 2798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2798 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2798 has memoization hash 7551f801457bcd5c430689c91e6a8793 parsl.dataflow.memoization INFO: Task 2798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2798 try 0 launched on executor multi with executor id 2799 parsl.dataflow.dflow INFO: Standard output for task 2798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/db1ff18b-89fc-417b-80ac-d22f2f5dabe1_isrForCrosstalkSources_1125891_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/db1ff18b-89fc-417b-80ac-d22f2f5dabe1_isrForCrosstalkSources_1125891_25.stderr parsl.dataflow.dflow DEBUG: Task 2799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2799 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2799 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2799 has memoization hash ad8f586bb4be709dd521cb63df9be4eb parsl.dataflow.memoization INFO: Task 2799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2799 try 0 launched on executor multi with executor id 2800 parsl.dataflow.dflow INFO: Standard output for task 2799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4797a256-d69e-426b-895c-edfd8170a487_isrForCrosstalkSources_1125893_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4797a256-d69e-426b-895c-edfd8170a487_isrForCrosstalkSources_1125893_30.stderr parsl.dataflow.dflow DEBUG: Task 2800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2800 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2800 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2800 has memoization hash ce98bd7a141a812a9aa4aa28d0cc0170 parsl.dataflow.memoization INFO: Task 2800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2800 try 0 launched on executor multi with executor id 2801 parsl.dataflow.dflow INFO: Standard output for task 2800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/ae06c30b-4bb9-45bf-a7d2-25d0b022c9d1_isrForCrosstalkSources_1125941_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/ae06c30b-4bb9-45bf-a7d2-25d0b022c9d1_isrForCrosstalkSources_1125941_46.stderr parsl.dataflow.dflow DEBUG: Task 2801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2801 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2801 has memoization hash 7cdd435694fa39a0a4b9dc24e21b739a parsl.dataflow.memoization INFO: Task 2801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2801 try 0 launched on executor multi with executor id 2802 parsl.dataflow.dflow INFO: Standard output for task 2801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/fe3d892e-80d6-472c-9438-53cc9ad93e55_isrForCrosstalkSources_1125901_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/fe3d892e-80d6-472c-9438-53cc9ad93e55_isrForCrosstalkSources_1125901_42.stderr parsl.dataflow.dflow DEBUG: Task 2802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2802 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2802 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2802 has memoization hash 6a8d6f8d366c8a088af6ad6beae8951a parsl.dataflow.memoization INFO: Task 2802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2802 try 0 launched on executor multi with executor id 2803 parsl.dataflow.dflow INFO: Standard output for task 2802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/908fd2b4-023a-472e-8e9f-ce96539c9e67_isrForCrosstalkSources_1125937_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/908fd2b4-023a-472e-8e9f-ce96539c9e67_isrForCrosstalkSources_1125937_8.stderr parsl.dataflow.dflow DEBUG: Task 2803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2803 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2803 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2803 has memoization hash 1c912d4fbfe19053beea715a4f474d06 parsl.dataflow.memoization INFO: Task 2803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2803 try 0 launched on executor multi with executor id 2804 parsl.dataflow.dflow INFO: Standard output for task 2803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/b306ac71-b40c-48f5-aa8f-14eaaf311028_isrForCrosstalkSources_1125891_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/b306ac71-b40c-48f5-aa8f-14eaaf311028_isrForCrosstalkSources_1125891_33.stderr parsl.dataflow.dflow DEBUG: Task 2804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2804 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2804 has memoization hash 38f9fdd1f42942f3d4d2eac60a45ea2d parsl.dataflow.memoization INFO: Task 2804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2804 try 0 launched on executor multi with executor id 2805 parsl.dataflow.dflow INFO: Standard output for task 2804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/798eaa2d-6545-4d53-ae12-8d8b502f6d48_isrForCrosstalkSources_1125943_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/798eaa2d-6545-4d53-ae12-8d8b502f6d48_isrForCrosstalkSources_1125943_57.stderr parsl.dataflow.dflow DEBUG: Task 2805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2805 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2805 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2805 has memoization hash 4db839bac8b2ffe44f2d7166b0703fb3 parsl.dataflow.memoization INFO: Task 2805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2805 try 0 launched on executor multi with executor id 2806 parsl.dataflow.dflow INFO: Standard output for task 2805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/f8c7d772-8a2a-436a-8d8a-8ddcd09d3c71_isrForCrosstalkSources_1125934_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/f8c7d772-8a2a-436a-8d8a-8ddcd09d3c71_isrForCrosstalkSources_1125934_23.stderr parsl.dataflow.dflow DEBUG: Task 2806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2806 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2806 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2806 has memoization hash ef281ac1ab180f5d01c95d83afedf229 parsl.dataflow.memoization INFO: Task 2806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2806 try 0 launched on executor multi with executor id 2807 parsl.dataflow.dflow INFO: Standard output for task 2806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/c007615b-df2f-452f-8bdd-c526d9f0ae59_isrForCrosstalkSources_1125907_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/c007615b-df2f-452f-8bdd-c526d9f0ae59_isrForCrosstalkSources_1125907_30.stderr parsl.dataflow.dflow DEBUG: Task 2807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2807 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2807 has memoization hash d9929f6f5d7c9e69cc78c2f3a83eccde parsl.dataflow.memoization INFO: Task 2807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2807 try 0 launched on executor multi with executor id 2808 parsl.dataflow.dflow INFO: Standard output for task 2807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/1cc5071b-86ec-4f8d-be84-9545622fdb8f_isrForCrosstalkSources_1125910_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/1cc5071b-86ec-4f8d-be84-9545622fdb8f_isrForCrosstalkSources_1125910_29.stderr parsl.dataflow.dflow DEBUG: Task 2808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2808 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2808 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2808 has memoization hash d6de9a69ec87da987bdcafcb4b7576b6 parsl.dataflow.memoization INFO: Task 2808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2808 try 0 launched on executor multi with executor id 2809 parsl.dataflow.dflow INFO: Standard output for task 2808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/42c770b0-36a6-45f9-ae54-bc3b51fb4824_isrForCrosstalkSources_1125922_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/42c770b0-36a6-45f9-ae54-bc3b51fb4824_isrForCrosstalkSources_1125922_53.stderr parsl.dataflow.dflow DEBUG: Task 2809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2809 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2809 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2809 has memoization hash 76d8768b39bed237d5ab0db44dd1371f parsl.dataflow.memoization INFO: Task 2809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2809 try 0 launched on executor multi with executor id 2810 parsl.dataflow.dflow INFO: Standard output for task 2809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/81fb1b82-8798-44ce-a588-50bca9de1173_isrForCrosstalkSources_1125902_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/81fb1b82-8798-44ce-a588-50bca9de1173_isrForCrosstalkSources_1125902_22.stderr parsl.dataflow.dflow DEBUG: Task 2810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2810 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2810 has memoization hash 68d3acada69a71e6102ca41fa2995b25 parsl.dataflow.memoization INFO: Task 2810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2810 try 0 launched on executor multi with executor id 2811 parsl.dataflow.dflow INFO: Standard output for task 2810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/23ac63f4-a4a1-4add-a71b-e16519afe112_isrForCrosstalkSources_1125931_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/23ac63f4-a4a1-4add-a71b-e16519afe112_isrForCrosstalkSources_1125931_48.stderr parsl.dataflow.dflow DEBUG: Task 2811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2811 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2811 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2811 has memoization hash 8dde4d59d4d80a953608a31ddc3ae619 parsl.dataflow.memoization INFO: Task 2811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2811 try 0 launched on executor multi with executor id 2812 parsl.dataflow.dflow INFO: Standard output for task 2811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2d5e6aab-2016-4a48-ae34-0e32951685b9_isrForCrosstalkSources_1125933_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2d5e6aab-2016-4a48-ae34-0e32951685b9_isrForCrosstalkSources_1125933_9.stderr parsl.dataflow.dflow DEBUG: Task 2812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2812 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2812 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2812 has memoization hash 67d35bdab223a01d9d111649ba830a04 parsl.dataflow.memoization INFO: Task 2812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2812 try 0 launched on executor multi with executor id 2813 parsl.dataflow.dflow INFO: Standard output for task 2812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/49f13f50-3ea7-4414-83c5-b282467fd757_isrForCrosstalkSources_1125896_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/49f13f50-3ea7-4414-83c5-b282467fd757_isrForCrosstalkSources_1125896_58.stderr parsl.dataflow.dflow DEBUG: Task 2813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2813 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2813 has memoization hash b0057d8073d0e8c8fb47d6f50f471b7f parsl.dataflow.memoization INFO: Task 2813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2813 try 0 launched on executor multi with executor id 2814 parsl.dataflow.dflow INFO: Standard output for task 2813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2a3d9367-e296-4d5e-b65a-d9ae832d0d0e_isrForCrosstalkSources_1125895_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2a3d9367-e296-4d5e-b65a-d9ae832d0d0e_isrForCrosstalkSources_1125895_42.stderr parsl.dataflow.dflow DEBUG: Task 2814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2814 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2814 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2814 has memoization hash 8dbbf9b3640b7f8ce15f60e2646371f1 parsl.dataflow.memoization INFO: Task 2814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2814 try 0 launched on executor multi with executor id 2815 parsl.dataflow.dflow INFO: Standard output for task 2814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/24bd5bed-b62a-4aa4-8082-9a5e5103e2e1_isrForCrosstalkSources_1125916_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/24bd5bed-b62a-4aa4-8082-9a5e5103e2e1_isrForCrosstalkSources_1125916_13.stderr parsl.dataflow.dflow DEBUG: Task 2815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2815 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2815 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2815 has memoization hash 8a3608fd74fdf24cbda83379ee5efb36 parsl.dataflow.memoization INFO: Task 2815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2815 try 0 launched on executor multi with executor id 2816 parsl.dataflow.dflow INFO: Standard output for task 2815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/23dc16b4-92a2-4953-9dff-1a767f2b4d6a_isrForCrosstalkSources_1125941_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/23dc16b4-92a2-4953-9dff-1a767f2b4d6a_isrForCrosstalkSources_1125941_24.stderr parsl.dataflow.dflow DEBUG: Task 2816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2816 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2816 has memoization hash 894364e1e6b595100913be153c5a453b parsl.dataflow.memoization INFO: Task 2816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2816 try 0 launched on executor multi with executor id 2817 parsl.dataflow.dflow INFO: Standard output for task 2816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/8dbf2f07-d29f-451b-bcbf-c8fbbdbbcf10_isrForCrosstalkSources_1125921_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/8dbf2f07-d29f-451b-bcbf-c8fbbdbbcf10_isrForCrosstalkSources_1125921_56.stderr parsl.dataflow.dflow DEBUG: Task 2817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2817 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2817 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2817 has memoization hash 504190f1a6d8ad5785291d2dd1f844bb parsl.dataflow.memoization INFO: Task 2817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2817 try 0 launched on executor multi with executor id 2818 parsl.dataflow.dflow INFO: Standard output for task 2817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ca7437f2-f9df-40e5-9fe5-7380398f70b3_isrForCrosstalkSources_1125902_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ca7437f2-f9df-40e5-9fe5-7380398f70b3_isrForCrosstalkSources_1125902_43.stderr parsl.dataflow.dflow DEBUG: Task 2818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2818 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2818 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2818 has memoization hash 82e71f9eec093a30662db99c3ce0798c parsl.dataflow.memoization INFO: Task 2818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:01 proc_lsst.multi:170 [INFO] [multi] job_id 16896426 proc_lsst.multi INFO: [multi] job_id 16896426 2024-03-07 15:07:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 15:07:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:01 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896426 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896426 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 16896426 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-07 15:07:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-07 15:07:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 15:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2818 try 0 launched on executor multi with executor id 2819 parsl.dataflow.dflow INFO: Standard output for task 2818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/7b863f91-b789-471e-b713-3ff087a3defe_isrForCrosstalkSources_1125944_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/7b863f91-b789-471e-b713-3ff087a3defe_isrForCrosstalkSources_1125944_47.stderr parsl.dataflow.dflow DEBUG: Task 2819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2819 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2819 has memoization hash e3cb04560980b516d06b4ebd255490e6 parsl.dataflow.memoization INFO: Task 2819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2819 try 0 launched on executor multi with executor id 2820 parsl.dataflow.dflow INFO: Standard output for task 2819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/be82cb31-c497-4678-82aa-0b663c29441b_isrForCrosstalkSources_1125904_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/be82cb31-c497-4678-82aa-0b663c29441b_isrForCrosstalkSources_1125904_40.stderr parsl.dataflow.dflow DEBUG: Task 2820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2820 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2820 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2820 has memoization hash f64f020e9b09dde135d7c71d9bd05512 parsl.dataflow.memoization INFO: Task 2820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2820 try 0 launched on executor multi with executor id 2821 parsl.dataflow.dflow INFO: Standard output for task 2820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/877618ef-44d7-4a1d-a991-5682b1e1c706_isrForCrosstalkSources_1125899_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/877618ef-44d7-4a1d-a991-5682b1e1c706_isrForCrosstalkSources_1125899_16.stderr parsl.dataflow.dflow DEBUG: Task 2821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2821 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2821 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2821 has memoization hash c84b9a2c44fb88330ba43bba5d3cadf9 parsl.dataflow.memoization INFO: Task 2821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2821 try 0 launched on executor multi with executor id 2822 parsl.dataflow.dflow INFO: Standard output for task 2821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/2b4ce0fe-e41e-4ac9-8269-57e21af1a1c1_isrForCrosstalkSources_1125905_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/2b4ce0fe-e41e-4ac9-8269-57e21af1a1c1_isrForCrosstalkSources_1125905_47.stderr parsl.dataflow.dflow DEBUG: Task 2822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2822 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2822 has memoization hash 478484f0eb762ae493e59fec0b2cfd22 parsl.dataflow.memoization INFO: Task 2822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396dba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2822 try 0 launched on executor multi with executor id 2823 parsl.dataflow.dflow INFO: Standard output for task 2822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/880e3b6a-446c-49fa-a8f5-6aa62c596821_isrForCrosstalkSources_1125905_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/880e3b6a-446c-49fa-a8f5-6aa62c596821_isrForCrosstalkSources_1125905_23.stderr parsl.dataflow.dflow DEBUG: Task 2823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2823 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2823 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2823 has memoization hash a13c84ec6db46e9aebac7f0fbb0000c3 parsl.dataflow.memoization INFO: Task 2823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2823 try 0 launched on executor multi with executor id 2824 parsl.dataflow.dflow INFO: Standard output for task 2823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7af73753-c781-4db1-8d7f-c1cfffa74a70_isrForCrosstalkSources_1125929_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7af73753-c781-4db1-8d7f-c1cfffa74a70_isrForCrosstalkSources_1125929_54.stderr parsl.dataflow.dflow DEBUG: Task 2824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2824 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2824 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2824 has memoization hash aac72a8d1cefc8abcdd1b73eea03c614 parsl.dataflow.memoization INFO: Task 2824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2824 try 0 launched on executor multi with executor id 2825 parsl.dataflow.dflow INFO: Standard output for task 2824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/22a2f76a-c1c2-4a99-9b17-5ef696328349_isrForCrosstalkSources_1125921_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/22a2f76a-c1c2-4a99-9b17-5ef696328349_isrForCrosstalkSources_1125921_8.stderr parsl.dataflow.dflow DEBUG: Task 2825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2825 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2825 has memoization hash c69529add055237012fe202e6597d149 parsl.dataflow.memoization INFO: Task 2825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2825 try 0 launched on executor multi with executor id 2826 parsl.dataflow.dflow INFO: Standard output for task 2825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/5b82f52d-fd6e-4a93-be8a-317a9fed036b_isrForCrosstalkSources_1125918_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/5b82f52d-fd6e-4a93-be8a-317a9fed036b_isrForCrosstalkSources_1125918_7.stderr parsl.dataflow.dflow DEBUG: Task 2826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2826 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2826 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2826 has memoization hash 115c8259c78d1a52dd9b9fac2b1e20e8 parsl.dataflow.memoization INFO: Task 2826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2826 try 0 launched on executor multi with executor id 2827 parsl.dataflow.dflow INFO: Standard output for task 2826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/4d33fd38-6ea5-4825-95c0-625a61dd56e2_isrForCrosstalkSources_1125918_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/4d33fd38-6ea5-4825-95c0-625a61dd56e2_isrForCrosstalkSources_1125918_2.stderr parsl.dataflow.dflow DEBUG: Task 2827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2827 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2827 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2827 has memoization hash c11023c67dfee9909f00596293cd9bec parsl.dataflow.memoization INFO: Task 2827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2827 try 0 launched on executor multi with executor id 2828 parsl.dataflow.dflow INFO: Standard output for task 2827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/82649a1e-c218-41ae-a121-6e9fcced4a8f_isrForCrosstalkSources_1125947_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/82649a1e-c218-41ae-a121-6e9fcced4a8f_isrForCrosstalkSources_1125947_15.stderr parsl.dataflow.dflow DEBUG: Task 2828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2828 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2828 has memoization hash 7876fb9b08965afcec2b1a0cad1ddb07 parsl.dataflow.memoization INFO: Task 2828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2828 try 0 launched on executor multi with executor id 2829 parsl.dataflow.dflow INFO: Standard output for task 2828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/963c37a9-fa0e-4bbd-8515-bc043ac4da14_isrForCrosstalkSources_1125920_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/963c37a9-fa0e-4bbd-8515-bc043ac4da14_isrForCrosstalkSources_1125920_9.stderr parsl.dataflow.dflow DEBUG: Task 2829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2829 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2829 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2829 has memoization hash 01897d556e6fe6664dccc31f109beeb1 parsl.dataflow.memoization INFO: Task 2829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2829 try 0 launched on executor multi with executor id 2830 parsl.dataflow.dflow INFO: Standard output for task 2829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/c10b9fb5-c528-4ce7-b86f-3c267abe70b8_isrForCrosstalkSources_1125888_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/c10b9fb5-c528-4ce7-b86f-3c267abe70b8_isrForCrosstalkSources_1125888_49.stderr parsl.dataflow.dflow DEBUG: Task 2830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2830 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2830 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2830 has memoization hash c9bd19d54850d473736d865b71619f27 parsl.dataflow.memoization INFO: Task 2830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2830 try 0 launched on executor multi with executor id 2831 parsl.dataflow.dflow INFO: Standard output for task 2830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/ad2de96f-5128-4ab6-a372-73af13bb824e_isrForCrosstalkSources_1125900_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/ad2de96f-5128-4ab6-a372-73af13bb824e_isrForCrosstalkSources_1125900_19.stderr parsl.dataflow.dflow DEBUG: Task 2831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2831 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2831 has memoization hash 39241c9e1965f521759a59f83dc298df parsl.dataflow.memoization INFO: Task 2831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2831 try 0 launched on executor multi with executor id 2832 parsl.dataflow.dflow INFO: Standard output for task 2831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/03bdb4ce-17b2-46d5-aaf7-93fb9e86c180_isrForCrosstalkSources_1125896_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/03bdb4ce-17b2-46d5-aaf7-93fb9e86c180_isrForCrosstalkSources_1125896_8.stderr parsl.dataflow.dflow DEBUG: Task 2832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2832 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2832 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2832 has memoization hash 8dc0bc5009c35e3e55c70050bb48536a parsl.dataflow.memoization INFO: Task 2832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2832 try 0 launched on executor multi with executor id 2833 parsl.dataflow.dflow INFO: Standard output for task 2832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d3a2674a-2ab6-40f6-b356-5600ff647046_isrForCrosstalkSources_1125911_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d3a2674a-2ab6-40f6-b356-5600ff647046_isrForCrosstalkSources_1125911_9.stderr parsl.dataflow.dflow DEBUG: Task 2833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2833 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2833 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2833 has memoization hash 3735deec2425ecd8090baf8ce4f1c9ae parsl.dataflow.memoization INFO: Task 2833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2833 try 0 launched on executor multi with executor id 2834 parsl.dataflow.dflow INFO: Standard output for task 2833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/abc71bb4-e246-40f4-8c1d-6741f98d1061_isrForCrosstalkSources_1125909_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/abc71bb4-e246-40f4-8c1d-6741f98d1061_isrForCrosstalkSources_1125909_8.stderr parsl.dataflow.dflow DEBUG: Task 2834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2834 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2834 has memoization hash 1ad64ad187d4f6b8dddba00ed200d289 parsl.dataflow.memoization INFO: Task 2834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2834 try 0 launched on executor multi with executor id 2835 parsl.dataflow.dflow INFO: Standard output for task 2834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/9f6dbb8d-7312-43e3-82b0-f0c4f3b118ef_isrForCrosstalkSources_1125891_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/9f6dbb8d-7312-43e3-82b0-f0c4f3b118ef_isrForCrosstalkSources_1125891_32.stderr parsl.dataflow.dflow DEBUG: Task 2835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2835 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2835 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2835 has memoization hash 1b0ee2818dadc5c3316aa626f0e79be8 parsl.dataflow.memoization INFO: Task 2835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2835 try 0 launched on executor multi with executor id 2836 parsl.dataflow.dflow INFO: Standard output for task 2835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/0dbc4433-0850-45c2-badb-0969c91b0a18_isrForCrosstalkSources_1125898_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/0dbc4433-0850-45c2-badb-0969c91b0a18_isrForCrosstalkSources_1125898_21.stderr parsl.dataflow.dflow DEBUG: Task 2836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2836 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2836 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2836 has memoization hash 603122e6db89fecfb60e9b21df128270 parsl.dataflow.memoization INFO: Task 2836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2836 try 0 launched on executor multi with executor id 2837 parsl.dataflow.dflow INFO: Standard output for task 2836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/c48c1af7-dc2b-40cc-a272-5f56194c119e_isrForCrosstalkSources_1125920_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/c48c1af7-dc2b-40cc-a272-5f56194c119e_isrForCrosstalkSources_1125920_60.stderr parsl.dataflow.dflow DEBUG: Task 2837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2837 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2837 has memoization hash fa521700e1711b0f137a1226fab306d9 parsl.dataflow.memoization INFO: Task 2837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2837 try 0 launched on executor multi with executor id 2838 parsl.dataflow.dflow INFO: Standard output for task 2837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/aa6a5575-1a4e-4ad5-af48-c9b53373215c_isrForCrosstalkSources_1125930_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/aa6a5575-1a4e-4ad5-af48-c9b53373215c_isrForCrosstalkSources_1125930_1.stderr parsl.dataflow.dflow DEBUG: Task 2838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2838 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2838 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2838 has memoization hash dcd9416dc2c470fe3ccdd79817d28239 parsl.dataflow.memoization INFO: Task 2838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2838 try 0 launched on executor multi with executor id 2839 parsl.dataflow.dflow INFO: Standard output for task 2838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/80a79cf1-37c5-4ccf-9e9f-8d831ccbd199_isrForCrosstalkSources_1125933_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/80a79cf1-37c5-4ccf-9e9f-8d831ccbd199_isrForCrosstalkSources_1125933_58.stderr parsl.dataflow.dflow DEBUG: Task 2839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2839 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2839 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2839 has memoization hash 09030799e1dbdd2949c79f3ec8676ba8 parsl.dataflow.memoization INFO: Task 2839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2839 try 0 launched on executor multi with executor id 2840 parsl.dataflow.dflow INFO: Standard output for task 2839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/0eca967e-8fd0-45ae-bfd9-771e6f06aab5_isrForCrosstalkSources_1125924_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/0eca967e-8fd0-45ae-bfd9-771e6f06aab5_isrForCrosstalkSources_1125924_52.stderr parsl.dataflow.dflow DEBUG: Task 2840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2840 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2840 has memoization hash 17bce79795571fdecf304486c847dc9c parsl.dataflow.memoization INFO: Task 2840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2840 try 0 launched on executor multi with executor id 2841 parsl.dataflow.dflow INFO: Standard output for task 2840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/b05112a2-3381-4640-8842-5adf5805c608_isrForCrosstalkSources_1125919_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/b05112a2-3381-4640-8842-5adf5805c608_isrForCrosstalkSources_1125919_56.stderr parsl.dataflow.dflow DEBUG: Task 2841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2841 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2841 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2841 has memoization hash 09acfa7c3f5527521790fe3ad836c84d parsl.dataflow.memoization INFO: Task 2841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2841 try 0 launched on executor multi with executor id 2842 parsl.dataflow.dflow INFO: Standard output for task 2841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/f716563d-37de-413e-b5a7-51b549d28e0b_isrForCrosstalkSources_1125903_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/f716563d-37de-413e-b5a7-51b549d28e0b_isrForCrosstalkSources_1125903_11.stderr parsl.dataflow.dflow DEBUG: Task 2842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2842 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2842 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2842 has memoization hash ab225eb32bd6148a508ee452fa387b84 parsl.dataflow.memoization INFO: Task 2842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2842 try 0 launched on executor multi with executor id 2843 parsl.dataflow.dflow INFO: Standard output for task 2842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/477571b4-a616-41f5-bcb7-140c10224880_isrForCrosstalkSources_1125910_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/477571b4-a616-41f5-bcb7-140c10224880_isrForCrosstalkSources_1125910_28.stderr parsl.dataflow.dflow DEBUG: Task 2843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2843 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2843 has memoization hash 45a50f8cc6f972b98dafff4c3b58bad8 parsl.dataflow.memoization INFO: Task 2843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2843 try 0 launched on executor multi with executor id 2844 parsl.dataflow.dflow INFO: Standard output for task 2843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c8d1d42f-d6eb-4bc1-946a-f888fc0e1a00_isrForCrosstalkSources_1125910_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c8d1d42f-d6eb-4bc1-946a-f888fc0e1a00_isrForCrosstalkSources_1125910_51.stderr parsl.dataflow.dflow DEBUG: Task 2844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2844 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2844 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2844 has memoization hash ecd9aed9c80c9eb14a5f9daa008f771b parsl.dataflow.memoization INFO: Task 2844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:01 proc_lsst.multi:170 [INFO] [multi] job_id 16896427 proc_lsst.multi INFO: [multi] job_id 16896427 2024-03-07 15:07:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 15:07:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:01 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896427 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896427 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 16896427 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-07 15:07:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-07 15:07:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 15:07:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2844 try 0 launched on executor multi with executor id 2845 parsl.dataflow.dflow INFO: Standard output for task 2844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/7318e62d-55c8-40b9-9022-e334c46b125a_isrForCrosstalkSources_1125902_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/7318e62d-55c8-40b9-9022-e334c46b125a_isrForCrosstalkSources_1125902_5.stderr parsl.dataflow.dflow DEBUG: Task 2845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2845 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2845 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2845 has memoization hash 323b1847fa8478aae57e7b587ecf0bc6 parsl.dataflow.memoization INFO: Task 2845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2845 try 0 launched on executor multi with executor id 2846 parsl.dataflow.dflow INFO: Standard output for task 2845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/a614c514-9dd0-4578-a580-5214ce782c57_isrForCrosstalkSources_1125892_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/a614c514-9dd0-4578-a580-5214ce782c57_isrForCrosstalkSources_1125892_3.stderr parsl.dataflow.dflow DEBUG: Task 2846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2846 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2846 has memoization hash 514517acf9de4ca3c8af325de9980b46 parsl.dataflow.memoization INFO: Task 2846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2846 try 0 launched on executor multi with executor id 2847 parsl.dataflow.dflow INFO: Standard output for task 2846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c0ab949b-cdfe-4f18-a69c-25181db83f50_isrForCrosstalkSources_1125893_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c0ab949b-cdfe-4f18-a69c-25181db83f50_isrForCrosstalkSources_1125893_57.stderr parsl.dataflow.dflow DEBUG: Task 2847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2847 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2847 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2847 has memoization hash 5a1dd66cbfeb61eaac5fee5304d2c2cf parsl.dataflow.memoization INFO: Task 2847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2847 try 0 launched on executor multi with executor id 2848 parsl.dataflow.dflow INFO: Standard output for task 2847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/5fdaa94e-e8c9-4839-a4d9-178df6523665_isrForCrosstalkSources_1125896_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/5fdaa94e-e8c9-4839-a4d9-178df6523665_isrForCrosstalkSources_1125896_56.stderr parsl.dataflow.dflow DEBUG: Task 2848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2848 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2848 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2848 has memoization hash a4170e9e9cc618a3be1cd9daa88b0e86 parsl.dataflow.memoization INFO: Task 2848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2848 try 0 launched on executor multi with executor id 2849 parsl.dataflow.dflow INFO: Standard output for task 2848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/52569885-3b08-4d50-ad0b-a0c5e691adb1_isrForCrosstalkSources_1125929_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/52569885-3b08-4d50-ad0b-a0c5e691adb1_isrForCrosstalkSources_1125929_36.stderr parsl.dataflow.dflow DEBUG: Task 2849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2849 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2849 has memoization hash dbc9526405b1deaeea52f02ae1730873 parsl.dataflow.memoization INFO: Task 2849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2849 try 0 launched on executor multi with executor id 2850 parsl.dataflow.dflow INFO: Standard output for task 2849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/043832e2-1119-4ca4-b880-1c23e6bd9324_isrForCrosstalkSources_1125914_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/043832e2-1119-4ca4-b880-1c23e6bd9324_isrForCrosstalkSources_1125914_19.stderr parsl.dataflow.dflow DEBUG: Task 2850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2850 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2850 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2850 has memoization hash 7ac6502095b08310b21eb4c3adb8b10b parsl.dataflow.memoization INFO: Task 2850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2850 try 0 launched on executor multi with executor id 2851 parsl.dataflow.dflow INFO: Standard output for task 2850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/16a21452-f209-46e1-b3ca-ef7d27354137_isrForCrosstalkSources_1125940_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/16a21452-f209-46e1-b3ca-ef7d27354137_isrForCrosstalkSources_1125940_5.stderr parsl.dataflow.dflow DEBUG: Task 2851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2851 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2851 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2851 has memoization hash 2bf9a0da6a60b4df4fd058847a6c6b8e parsl.dataflow.memoization INFO: Task 2851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2851 try 0 launched on executor multi with executor id 2852 parsl.dataflow.dflow INFO: Standard output for task 2851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/640b5a39-2772-49b8-8de3-dfe4121c8a6e_isrForCrosstalkSources_1125899_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/640b5a39-2772-49b8-8de3-dfe4121c8a6e_isrForCrosstalkSources_1125899_30.stderr parsl.dataflow.dflow DEBUG: Task 2852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2852 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2852 has memoization hash 58e79ab67886c69e2a17e56ded08485a parsl.dataflow.memoization INFO: Task 2852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2852 try 0 launched on executor multi with executor id 2853 parsl.dataflow.dflow INFO: Standard output for task 2852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/8c76dc9b-38bf-4cad-ace2-a51718e572bf_isrForCrosstalkSources_1125906_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/8c76dc9b-38bf-4cad-ace2-a51718e572bf_isrForCrosstalkSources_1125906_12.stderr parsl.dataflow.dflow DEBUG: Task 2853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2853 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2853 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2853 has memoization hash cf46bd187b04d17fdbc8f00024233562 parsl.dataflow.memoization INFO: Task 2853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2853 try 0 launched on executor multi with executor id 2854 parsl.dataflow.dflow INFO: Standard output for task 2853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/a93bb9d6-7b00-481a-9ea6-b6b37b2dd753_isrForCrosstalkSources_1125943_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/a93bb9d6-7b00-481a-9ea6-b6b37b2dd753_isrForCrosstalkSources_1125943_40.stderr parsl.dataflow.dflow DEBUG: Task 2854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2854 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2854 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2854 has memoization hash 986a7f6fbc3b80ab37bafd330748319b parsl.dataflow.memoization INFO: Task 2854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2854 try 0 launched on executor multi with executor id 2855 parsl.dataflow.dflow INFO: Standard output for task 2854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/bd4d8fc4-8c14-4114-86fe-e6ef16f88f85_isrForCrosstalkSources_1125888_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/bd4d8fc4-8c14-4114-86fe-e6ef16f88f85_isrForCrosstalkSources_1125888_25.stderr parsl.dataflow.dflow DEBUG: Task 2855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2855 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2855 has memoization hash 7ae36d45b66f6c8f4973838581bee80a parsl.dataflow.memoization INFO: Task 2855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2855 try 0 launched on executor multi with executor id 2856 parsl.dataflow.dflow INFO: Standard output for task 2855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1e5af2d0-573a-4428-98d3-d872ca8cb70d_isrForCrosstalkSources_1125900_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1e5af2d0-573a-4428-98d3-d872ca8cb70d_isrForCrosstalkSources_1125900_42.stderr parsl.dataflow.dflow DEBUG: Task 2856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2856 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2856 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2856 has memoization hash 1bf6a868c07a7d1c72882e9059fa35e2 parsl.dataflow.memoization INFO: Task 2856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2856 try 0 launched on executor multi with executor id 2857 parsl.dataflow.dflow INFO: Standard output for task 2856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/4db929b6-cb68-45e3-abc7-bda5ed82892f_isrForCrosstalkSources_1125945_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/4db929b6-cb68-45e3-abc7-bda5ed82892f_isrForCrosstalkSources_1125945_45.stderr parsl.dataflow.dflow DEBUG: Task 2857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2857 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2857 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2857 has memoization hash 446dd8d5e013ec39eb632f03d18e6e8e parsl.dataflow.memoization INFO: Task 2857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2857 try 0 launched on executor multi with executor id 2858 parsl.dataflow.dflow INFO: Standard output for task 2857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/00a5bf29-afcf-4ee4-b36a-26d57c67a142_isrForCrosstalkSources_1125940_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/00a5bf29-afcf-4ee4-b36a-26d57c67a142_isrForCrosstalkSources_1125940_8.stderr parsl.dataflow.dflow DEBUG: Task 2858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2858 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2858 has memoization hash 69e36ca324643f4438ed80095406dba6 parsl.dataflow.memoization INFO: Task 2858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2858 try 0 launched on executor multi with executor id 2859 parsl.dataflow.dflow INFO: Standard output for task 2858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/fc221f2b-3c3a-4fda-9486-999157483956_isrForCrosstalkSources_1125901_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/fc221f2b-3c3a-4fda-9486-999157483956_isrForCrosstalkSources_1125901_36.stderr parsl.dataflow.dflow DEBUG: Task 2859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2859 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2859 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2859 has memoization hash 2b894db83751f9c486758b32a44d817a parsl.dataflow.memoization INFO: Task 2859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2859 try 0 launched on executor multi with executor id 2860 parsl.dataflow.dflow INFO: Standard output for task 2859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/1a5a4af6-1538-4e07-9d6c-b1dcc0fa6827_isrForCrosstalkSources_1125913_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/1a5a4af6-1538-4e07-9d6c-b1dcc0fa6827_isrForCrosstalkSources_1125913_30.stderr parsl.dataflow.dflow DEBUG: Task 2860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2860 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2860 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2860 has memoization hash ca3c63963154fc513999d5ab58f17c54 parsl.dataflow.memoization INFO: Task 2860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2860 try 0 launched on executor multi with executor id 2861 parsl.dataflow.dflow INFO: Standard output for task 2860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/a0cd4dce-eb29-4368-9ec7-a289ccd13cff_isrForCrosstalkSources_1125899_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/a0cd4dce-eb29-4368-9ec7-a289ccd13cff_isrForCrosstalkSources_1125899_3.stderr parsl.dataflow.dflow DEBUG: Task 2861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2861 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2861 has memoization hash 79f46d6c8858c2d98ec4a5e2b76b07db parsl.dataflow.memoization INFO: Task 2861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2861 try 0 launched on executor multi with executor id 2862 parsl.dataflow.dflow INFO: Standard output for task 2861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/22960df5-fd80-4a2c-9968-f9a1cda3a6ff_isrForCrosstalkSources_1125910_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/22960df5-fd80-4a2c-9968-f9a1cda3a6ff_isrForCrosstalkSources_1125910_21.stderr parsl.dataflow.dflow DEBUG: Task 2862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2862 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2862 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2862 has memoization hash ca58c864926692f9b58ff3656b87733e parsl.dataflow.memoization INFO: Task 2862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2862 try 0 launched on executor multi with executor id 2863 parsl.dataflow.dflow INFO: Standard output for task 2862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/15e16e1e-dcd4-4f7a-a74e-86f487243c95_isrForCrosstalkSources_1125891_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/15e16e1e-dcd4-4f7a-a74e-86f487243c95_isrForCrosstalkSources_1125891_42.stderr parsl.dataflow.dflow DEBUG: Task 2863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2863 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2863 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2863 has memoization hash 016d32f7e418d25db65eb66e8f9cd9df parsl.dataflow.memoization INFO: Task 2863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2863 try 0 launched on executor multi with executor id 2864 parsl.dataflow.dflow INFO: Standard output for task 2863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/a7feb9d5-d6cf-4249-b2b2-8084ce9c85fc_isrForCrosstalkSources_1125922_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/a7feb9d5-d6cf-4249-b2b2-8084ce9c85fc_isrForCrosstalkSources_1125922_40.stderr parsl.dataflow.dflow DEBUG: Task 2864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2864 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2864 has memoization hash 76d1a3265ca5acc95b9f86d0edd1056d parsl.dataflow.memoization INFO: Task 2864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2864 try 0 launched on executor multi with executor id 2865 parsl.dataflow.dflow INFO: Standard output for task 2864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/815304be-8498-4739-bd5f-4af221b21e86_isrForCrosstalkSources_1125908_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/815304be-8498-4739-bd5f-4af221b21e86_isrForCrosstalkSources_1125908_40.stderr parsl.dataflow.dflow DEBUG: Task 2865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2865 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2865 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2865 has memoization hash 9e181a7daa2d99a2de5425647ad7a1ca parsl.dataflow.memoization INFO: Task 2865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2865 try 0 launched on executor multi with executor id 2866 parsl.dataflow.dflow INFO: Standard output for task 2865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/4d1b0e6f-f99b-4f29-8215-5c4fc99dff16_isrForCrosstalkSources_1125899_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/4d1b0e6f-f99b-4f29-8215-5c4fc99dff16_isrForCrosstalkSources_1125899_42.stderr parsl.dataflow.dflow DEBUG: Task 2866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2866 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2866 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2866 has memoization hash 1d95a0f4c383d3cb4c415809889828a6 parsl.dataflow.memoization INFO: Task 2866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2866 try 0 launched on executor multi with executor id 2867 parsl.dataflow.dflow INFO: Standard output for task 2866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/f91ad774-67fe-4513-9cd6-8e8d5ca9af9b_isrForCrosstalkSources_1125896_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/f91ad774-67fe-4513-9cd6-8e8d5ca9af9b_isrForCrosstalkSources_1125896_23.stderr parsl.dataflow.dflow DEBUG: Task 2867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2867 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2867 has memoization hash f31ee21502953ab0cc8bf295c79a2d02 parsl.dataflow.memoization INFO: Task 2867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2867 try 0 launched on executor multi with executor id 2868 parsl.dataflow.dflow INFO: Standard output for task 2867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6af75b3b-c7fe-4186-a25c-e6fcb5c4ec2a_isrForCrosstalkSources_1125927_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6af75b3b-c7fe-4186-a25c-e6fcb5c4ec2a_isrForCrosstalkSources_1125927_12.stderr 2024-03-07 15:07:02 proc_lsst.multi:170 [INFO] [multi] job_id 16896428 proc_lsst.multi INFO: [multi] job_id 16896428 2024-03-07 15:07:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:02 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896428 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896428 parsl.dataflow.dflow DEBUG: Task 2868 will be sent to executor multi parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 16896428 parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.executors.status_handling INFO: Allocated block ID 41 parsl.dataflow.dflow INFO: Task 2868 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-07 15:07:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-07 15:07:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.dataflow.dflow DEBUG: Task 2868 set to pending state with AppFuture: parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2868 has memoization hash f9f508c1b351d20cba651c2349798654 parsl.dataflow.memoization INFO: Task 2868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2868 try 0 launched on executor multi with executor id 2869 parsl.dataflow.dflow INFO: Standard output for task 2868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/fe5881cc-a96e-42f4-a662-3c287a1c6096_isrForCrosstalkSources_1125926_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/fe5881cc-a96e-42f4-a662-3c287a1c6096_isrForCrosstalkSources_1125926_55.stderr parsl.dataflow.dflow DEBUG: Task 2869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2869 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2869 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2869 has memoization hash 58cb8a14b9b98c5ab2bb766ed03500de parsl.dataflow.memoization INFO: Task 2869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2869 try 0 launched on executor multi with executor id 2870 parsl.dataflow.dflow INFO: Standard output for task 2869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/0e73b55a-33e0-4abf-b6c9-6ae636c8a27d_isrForCrosstalkSources_1125947_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/0e73b55a-33e0-4abf-b6c9-6ae636c8a27d_isrForCrosstalkSources_1125947_13.stderr parsl.dataflow.dflow DEBUG: Task 2870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2870 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2870 has memoization hash 1fa37f56a4cc8847d092ce9b819a1371 parsl.dataflow.memoization INFO: Task 2870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2870 try 0 launched on executor multi with executor id 2871 parsl.dataflow.dflow INFO: Standard output for task 2870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/f0d5cd0b-b5f6-45ec-8ace-b276603ea8df_isrForCrosstalkSources_1125941_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/f0d5cd0b-b5f6-45ec-8ace-b276603ea8df_isrForCrosstalkSources_1125941_2.stderr parsl.dataflow.dflow DEBUG: Task 2871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2871 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2871 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2871 has memoization hash fcbaa0b0bfba6c122ee79c01ee1fe22c parsl.dataflow.memoization INFO: Task 2871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3941f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2871 try 0 launched on executor multi with executor id 2872 parsl.dataflow.dflow INFO: Standard output for task 2871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/ba7af326-ed82-4f92-a037-fc507024dbbf_isrForCrosstalkSources_1125911_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/ba7af326-ed82-4f92-a037-fc507024dbbf_isrForCrosstalkSources_1125911_16.stderr parsl.dataflow.dflow DEBUG: Task 2872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2872 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2872 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2872 has memoization hash c4700e9847ee1f831ffe013855a517f0 parsl.dataflow.memoization INFO: Task 2872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2872 try 0 launched on executor multi with executor id 2873 parsl.dataflow.dflow INFO: Standard output for task 2872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/6e0d7666-85ab-4868-96a7-b5d461503522_isrForCrosstalkSources_1125915_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/6e0d7666-85ab-4868-96a7-b5d461503522_isrForCrosstalkSources_1125915_52.stderr parsl.dataflow.dflow DEBUG: Task 2873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2873 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2873 has memoization hash e1320fda605fea71a1b986f79c500ffb parsl.dataflow.memoization INFO: Task 2873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2873 try 0 launched on executor multi with executor id 2874 parsl.dataflow.dflow INFO: Standard output for task 2873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/269bc177-032e-4bf1-921c-3974acae78ee_isrForCrosstalkSources_1125903_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/269bc177-032e-4bf1-921c-3974acae78ee_isrForCrosstalkSources_1125903_21.stderr parsl.dataflow.dflow DEBUG: Task 2874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2874 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2874 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2874 has memoization hash 1677bdc651611a4388b79e9ce31488b0 parsl.dataflow.memoization INFO: Task 2874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2874 try 0 launched on executor multi with executor id 2875 parsl.dataflow.dflow INFO: Standard output for task 2874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/ace705b0-afba-420e-9f16-0157e47ec7e2_isrForCrosstalkSources_1125925_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/ace705b0-afba-420e-9f16-0157e47ec7e2_isrForCrosstalkSources_1125925_5.stderr parsl.dataflow.dflow DEBUG: Task 2875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2875 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2875 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2875 has memoization hash db4a95ac85ced6fa30197d6a7ceb3901 parsl.dataflow.memoization INFO: Task 2875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2875 try 0 launched on executor multi with executor id 2876 parsl.dataflow.dflow INFO: Standard output for task 2875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/5923154c-2c1e-4748-9378-e1e1ca317e2a_isrForCrosstalkSources_1125896_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/5923154c-2c1e-4748-9378-e1e1ca317e2a_isrForCrosstalkSources_1125896_36.stderr parsl.dataflow.dflow DEBUG: Task 2876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2876 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2876 has memoization hash c941bfba016de19251be852d47706215 parsl.dataflow.memoization INFO: Task 2876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2876 try 0 launched on executor multi with executor id 2877 parsl.dataflow.dflow INFO: Standard output for task 2876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d6210e4b-cfda-4975-91ad-e372681d0ca8_isrForCrosstalkSources_1125890_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d6210e4b-cfda-4975-91ad-e372681d0ca8_isrForCrosstalkSources_1125890_58.stderr parsl.dataflow.dflow DEBUG: Task 2877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2877 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2877 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2877 has memoization hash 416d7223a62caba21733002e52ff3f11 parsl.dataflow.memoization INFO: Task 2877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2877 try 0 launched on executor multi with executor id 2878 parsl.dataflow.dflow INFO: Standard output for task 2877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/f97ad47c-2bb9-4016-a25c-313d00aeb79b_isrForCrosstalkSources_1125921_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/f97ad47c-2bb9-4016-a25c-313d00aeb79b_isrForCrosstalkSources_1125921_36.stderr parsl.dataflow.dflow DEBUG: Task 2878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2878 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2878 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2878 has memoization hash de14f9490932be94abb5cc9d69e0fc6d parsl.dataflow.memoization INFO: Task 2878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2878 try 0 launched on executor multi with executor id 2879 parsl.dataflow.dflow INFO: Standard output for task 2878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/3b4dfc17-edd3-4611-b127-d85ed31f07f3_isrForCrosstalkSources_1125914_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/3b4dfc17-edd3-4611-b127-d85ed31f07f3_isrForCrosstalkSources_1125914_29.stderr parsl.dataflow.dflow DEBUG: Task 2879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2879 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2879 has memoization hash 5f5ba6b596c93ade6decb820e3e61881 parsl.dataflow.memoization INFO: Task 2879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2879 try 0 launched on executor multi with executor id 2880 parsl.dataflow.dflow INFO: Standard output for task 2879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/55d47737-9b9e-4eb9-8005-1a260c0f22af_isrForCrosstalkSources_1125909_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/55d47737-9b9e-4eb9-8005-1a260c0f22af_isrForCrosstalkSources_1125909_39.stderr parsl.dataflow.dflow DEBUG: Task 2880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2880 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2880 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2880 has memoization hash 099e86df00eb6cde7ad3f007c290cb52 parsl.dataflow.memoization INFO: Task 2880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2880 try 0 launched on executor multi with executor id 2881 parsl.dataflow.dflow INFO: Standard output for task 2880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/a139a535-8055-4dc8-a004-83dd7b654a25_isrForCrosstalkSources_1125932_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/a139a535-8055-4dc8-a004-83dd7b654a25_isrForCrosstalkSources_1125932_25.stderr parsl.dataflow.dflow DEBUG: Task 2881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2881 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2881 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2881 has memoization hash a9b81ce6454a1fd4e0766898e787a0e5 parsl.dataflow.memoization INFO: Task 2881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2881 try 0 launched on executor multi with executor id 2882 parsl.dataflow.dflow INFO: Standard output for task 2881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/b81ac122-0fb4-44e9-b9b5-584c83a39119_isrForCrosstalkSources_1125893_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/b81ac122-0fb4-44e9-b9b5-584c83a39119_isrForCrosstalkSources_1125893_11.stderr parsl.dataflow.dflow DEBUG: Task 2882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2882 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2882 has memoization hash 4dacd8ed5668897232b8099ad3b8a8c4 parsl.dataflow.memoization INFO: Task 2882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2882 try 0 launched on executor multi with executor id 2883 parsl.dataflow.dflow INFO: Standard output for task 2882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/06a1204a-9bfe-46e1-90d7-555e6300e18b_isrForCrosstalkSources_1125899_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/06a1204a-9bfe-46e1-90d7-555e6300e18b_isrForCrosstalkSources_1125899_31.stderr parsl.dataflow.dflow DEBUG: Task 2883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2883 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2883 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2883 has memoization hash ada39ea95e2177356f39bbfa20fcb51c parsl.dataflow.memoization INFO: Task 2883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2883 try 0 launched on executor multi with executor id 2884 parsl.dataflow.dflow INFO: Standard output for task 2883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/b1d993ea-f1aa-4522-9b8f-24e764be4f59_isrForCrosstalkSources_1125935_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/b1d993ea-f1aa-4522-9b8f-24e764be4f59_isrForCrosstalkSources_1125935_56.stderr parsl.dataflow.dflow DEBUG: Task 2884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2884 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2884 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2884 has memoization hash 8512e89e944a8b900c7a8ac3d4058ce6 parsl.dataflow.memoization INFO: Task 2884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2884 try 0 launched on executor multi with executor id 2885 parsl.dataflow.dflow INFO: Standard output for task 2884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/b0ff2c2b-b6b3-4e34-b5a0-a8f0d63086cc_isrForCrosstalkSources_1125895_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/b0ff2c2b-b6b3-4e34-b5a0-a8f0d63086cc_isrForCrosstalkSources_1125895_26.stderr parsl.dataflow.dflow DEBUG: Task 2885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2885 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2885 has memoization hash b4624a739e6d8ed01e6210b6c6edba95 parsl.dataflow.memoization INFO: Task 2885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2885 try 0 launched on executor multi with executor id 2886 parsl.dataflow.dflow INFO: Standard output for task 2885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/8a106361-b393-42a0-8072-f9f96de6c379_isrForCrosstalkSources_1125941_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/8a106361-b393-42a0-8072-f9f96de6c379_isrForCrosstalkSources_1125941_10.stderr parsl.dataflow.dflow DEBUG: Task 2886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2886 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2886 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2886 has memoization hash 87f9e542ae702e8a572a16a6a60fd0df parsl.dataflow.memoization INFO: Task 2886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2886 try 0 launched on executor multi with executor id 2887 parsl.dataflow.dflow INFO: Standard output for task 2886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/0f29c58c-7d9f-4b6f-9d53-bb966b273d21_isrForCrosstalkSources_1125943_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/0f29c58c-7d9f-4b6f-9d53-bb966b273d21_isrForCrosstalkSources_1125943_25.stderr parsl.dataflow.dflow DEBUG: Task 2887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2887 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2887 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2887 has memoization hash 2033b12d9f18e762af042878db969533 parsl.dataflow.memoization INFO: Task 2887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2887 try 0 launched on executor multi with executor id 2888 parsl.dataflow.dflow INFO: Standard output for task 2887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/0ff04c2e-fc01-45d8-a6b3-ddd3823e49d9_isrForCrosstalkSources_1125926_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/0ff04c2e-fc01-45d8-a6b3-ddd3823e49d9_isrForCrosstalkSources_1125926_59.stderr parsl.dataflow.dflow DEBUG: Task 2888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2888 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2888 has memoization hash 5bdda417e351d1d6fddb5375f743463c parsl.dataflow.memoization INFO: Task 2888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2888 try 0 launched on executor multi with executor id 2889 parsl.dataflow.dflow INFO: Standard output for task 2888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/4a3b9d5d-c750-4aa6-93b0-da1777c8a35b_isrForCrosstalkSources_1125928_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/4a3b9d5d-c750-4aa6-93b0-da1777c8a35b_isrForCrosstalkSources_1125928_16.stderr parsl.dataflow.dflow DEBUG: Task 2889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2889 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2889 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2889 has memoization hash 36325479f3a0bcee4ffc9b3bc5493728 parsl.dataflow.memoization INFO: Task 2889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2889 try 0 launched on executor multi with executor id 2890 parsl.dataflow.dflow INFO: Standard output for task 2889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1841d4f0-3d33-4481-983a-96e495883490_isrForCrosstalkSources_1125900_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1841d4f0-3d33-4481-983a-96e495883490_isrForCrosstalkSources_1125900_15.stderr parsl.dataflow.dflow DEBUG: Task 2890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2890 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2890 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2890 has memoization hash 9694228f4ab785168a1f1bd0674fac1e parsl.dataflow.memoization INFO: Task 2890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2890 try 0 launched on executor multi with executor id 2891 parsl.dataflow.dflow INFO: Standard output for task 2890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/daf09fbb-9127-4d24-aaf5-d25509fd4f4b_isrForCrosstalkSources_1125888_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/daf09fbb-9127-4d24-aaf5-d25509fd4f4b_isrForCrosstalkSources_1125888_26.stderr parsl.dataflow.dflow DEBUG: Task 2891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2891 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2891 has memoization hash 982af632f66800c5280151042ee5877b parsl.dataflow.memoization INFO: Task 2891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2891 try 0 launched on executor multi with executor id 2892 parsl.dataflow.dflow INFO: Standard output for task 2891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/884c5c08-a1f1-42a3-b111-fbda77c0e36a_isrForCrosstalkSources_1125902_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/884c5c08-a1f1-42a3-b111-fbda77c0e36a_isrForCrosstalkSources_1125902_54.stderr parsl.dataflow.dflow DEBUG: Task 2892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2892 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2892 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2892 has memoization hash 3318064ff7292524e2a75436587c7f01 parsl.dataflow.memoization INFO: Task 2892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:02 proc_lsst.multi:170 [INFO] [multi] job_id 16896429 proc_lsst.multi INFO: [multi] job_id 16896429 2024-03-07 15:07:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 15:07:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:02 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896429 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896429 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 16896429 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-07 15:07:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-07 15:07:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 15:07:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2892 try 0 launched on executor multi with executor id 2893 parsl.dataflow.dflow INFO: Standard output for task 2892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/8e7fdf66-a079-4e59-83a0-b2a763a845bc_isrForCrosstalkSources_1125933_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/8e7fdf66-a079-4e59-83a0-b2a763a845bc_isrForCrosstalkSources_1125933_54.stderr parsl.dataflow.dflow DEBUG: Task 2893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2893 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2893 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2893 has memoization hash 531f8395ff01c0bcda172c73b756fa84 parsl.dataflow.memoization INFO: Task 2893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2893 try 0 launched on executor multi with executor id 2894 parsl.dataflow.dflow INFO: Standard output for task 2893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/49505597-b35a-4e0f-a89a-b35f6c878822_isrForCrosstalkSources_1125924_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/49505597-b35a-4e0f-a89a-b35f6c878822_isrForCrosstalkSources_1125924_45.stderr parsl.dataflow.dflow DEBUG: Task 2894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2894 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2894 has memoization hash aa4bad756c448b38e2afa4afd39e0968 parsl.dataflow.memoization INFO: Task 2894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2894 try 0 launched on executor multi with executor id 2895 parsl.dataflow.dflow INFO: Standard output for task 2894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/517af224-5c61-4310-b89c-790e52434017_isrForCrosstalkSources_1125934_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/517af224-5c61-4310-b89c-790e52434017_isrForCrosstalkSources_1125934_32.stderr parsl.dataflow.dflow DEBUG: Task 2895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2895 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2895 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2895 has memoization hash 44988f2641c4771364f38c9243fe2c4d parsl.dataflow.memoization INFO: Task 2895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2895 try 0 launched on executor multi with executor id 2896 parsl.dataflow.dflow INFO: Standard output for task 2895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/52c1fab9-86e6-4188-af84-ade5e79e6c59_isrForCrosstalkSources_1125939_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/52c1fab9-86e6-4188-af84-ade5e79e6c59_isrForCrosstalkSources_1125939_41.stderr parsl.dataflow.dflow DEBUG: Task 2896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2896 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2896 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2896 has memoization hash 1bb120c074271626fc283b8f1ad89b74 parsl.dataflow.memoization INFO: Task 2896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2896 try 0 launched on executor multi with executor id 2897 parsl.dataflow.dflow INFO: Standard output for task 2896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/1e0a29a6-7d9e-4b0f-a440-530fbf502033_isrForCrosstalkSources_1125923_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/1e0a29a6-7d9e-4b0f-a440-530fbf502033_isrForCrosstalkSources_1125923_49.stderr parsl.dataflow.dflow DEBUG: Task 2897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2897 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2897 has memoization hash f4e79bad48d2baf5691d16e5cbdcbc41 parsl.dataflow.memoization INFO: Task 2897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2897 try 0 launched on executor multi with executor id 2898 parsl.dataflow.dflow INFO: Standard output for task 2897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/8aa8e045-368d-4133-a968-ee86c5ac2efa_isrForCrosstalkSources_1125917_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/8aa8e045-368d-4133-a968-ee86c5ac2efa_isrForCrosstalkSources_1125917_31.stderr parsl.dataflow.dflow DEBUG: Task 2898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2898 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2898 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2898 has memoization hash 88a885fe5afe0d25c135866a2934570c parsl.dataflow.memoization INFO: Task 2898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2898 try 0 launched on executor multi with executor id 2899 parsl.dataflow.dflow INFO: Standard output for task 2898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/31c3f5df-2535-4faa-9cf6-573fbec29b5c_isrForCrosstalkSources_1125941_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/31c3f5df-2535-4faa-9cf6-573fbec29b5c_isrForCrosstalkSources_1125941_9.stderr parsl.dataflow.dflow DEBUG: Task 2899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2899 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2899 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2899 has memoization hash 638b10a3d1e6578c56002fd419219338 parsl.dataflow.memoization INFO: Task 2899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2899 try 0 launched on executor multi with executor id 2900 parsl.dataflow.dflow INFO: Standard output for task 2899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/69475c1c-e938-4fcc-a5cf-f0116c31da3e_isrForCrosstalkSources_1125907_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/69475c1c-e938-4fcc-a5cf-f0116c31da3e_isrForCrosstalkSources_1125907_43.stderr parsl.dataflow.dflow DEBUG: Task 2900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2900 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2900 has memoization hash 3fa0654649b0f6e5aae9b3fc3a333e51 parsl.dataflow.memoization INFO: Task 2900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2900 try 0 launched on executor multi with executor id 2901 parsl.dataflow.dflow INFO: Standard output for task 2900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a6bbeb51-b0fa-4755-8ec1-97c618bdbdea_isrForCrosstalkSources_1125909_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a6bbeb51-b0fa-4755-8ec1-97c618bdbdea_isrForCrosstalkSources_1125909_60.stderr parsl.dataflow.dflow DEBUG: Task 2901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2901 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2901 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2901 has memoization hash ef2e004e22475d5ddba993efbfd53c3f parsl.dataflow.memoization INFO: Task 2901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2901 try 0 launched on executor multi with executor id 2902 parsl.dataflow.dflow INFO: Standard output for task 2901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/7d939404-1755-41ee-8691-8acb596f492e_isrForCrosstalkSources_1125938_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/7d939404-1755-41ee-8691-8acb596f492e_isrForCrosstalkSources_1125938_20.stderr parsl.dataflow.dflow DEBUG: Task 2902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2902 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2902 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2902 has memoization hash 8aa555d98cf9240ae3699296d4cb160f parsl.dataflow.memoization INFO: Task 2902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2902 try 0 launched on executor multi with executor id 2903 parsl.dataflow.dflow INFO: Standard output for task 2902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/81e5435c-b020-4c30-85e5-b99e608a371d_isrForCrosstalkSources_1125937_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/81e5435c-b020-4c30-85e5-b99e608a371d_isrForCrosstalkSources_1125937_29.stderr parsl.dataflow.dflow DEBUG: Task 2903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2903 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2903 has memoization hash b8acda4668c4aa01dc7cf42246947bb9 parsl.dataflow.memoization INFO: Task 2903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2903 try 0 launched on executor multi with executor id 2904 parsl.dataflow.dflow INFO: Standard output for task 2903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/cb9d45df-464c-42c0-a016-b0dcfbdc028d_isrForCrosstalkSources_1125910_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/cb9d45df-464c-42c0-a016-b0dcfbdc028d_isrForCrosstalkSources_1125910_19.stderr parsl.dataflow.dflow DEBUG: Task 2904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2904 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2904 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2904 has memoization hash af90666afb3e4144a2182cbfe5c93365 parsl.dataflow.memoization INFO: Task 2904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2904 try 0 launched on executor multi with executor id 2905 parsl.dataflow.dflow INFO: Standard output for task 2904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/df68f3a7-5701-4311-bcfb-749e17dfa000_isrForCrosstalkSources_1125922_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/df68f3a7-5701-4311-bcfb-749e17dfa000_isrForCrosstalkSources_1125922_19.stderr parsl.dataflow.dflow DEBUG: Task 2905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2905 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2905 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2905 has memoization hash 5e68885b46b96d50d357af7eeb59204d parsl.dataflow.memoization INFO: Task 2905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2905 try 0 launched on executor multi with executor id 2906 parsl.dataflow.dflow INFO: Standard output for task 2905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/6db1bb92-92a0-4d16-b279-32ff005a6ff6_isrForCrosstalkSources_1125900_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/6db1bb92-92a0-4d16-b279-32ff005a6ff6_isrForCrosstalkSources_1125900_34.stderr parsl.dataflow.dflow DEBUG: Task 2906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2906 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2906 has memoization hash 0c74a5edb813448b03ab40c20177a8d7 parsl.dataflow.memoization INFO: Task 2906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2906 try 0 launched on executor multi with executor id 2907 parsl.dataflow.dflow INFO: Standard output for task 2906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/acb8d237-f6ae-4e57-834a-04f8064e25e1_isrForCrosstalkSources_1125905_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/acb8d237-f6ae-4e57-834a-04f8064e25e1_isrForCrosstalkSources_1125905_8.stderr parsl.dataflow.dflow DEBUG: Task 2907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2907 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2907 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2907 has memoization hash 1075bedc5feef3553dc0d93acf362b7f parsl.dataflow.memoization INFO: Task 2907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2907 try 0 launched on executor multi with executor id 2908 parsl.dataflow.dflow INFO: Standard output for task 2907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/ec04d94f-a830-4d9f-85f3-9c195aa4cb91_isrForCrosstalkSources_1125935_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/ec04d94f-a830-4d9f-85f3-9c195aa4cb91_isrForCrosstalkSources_1125935_48.stderr parsl.dataflow.dflow DEBUG: Task 2908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2908 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2908 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2908 has memoization hash a7f6005bc9473d0ac3c4001fc66856f8 parsl.dataflow.memoization INFO: Task 2908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2908 try 0 launched on executor multi with executor id 2909 parsl.dataflow.dflow INFO: Standard output for task 2908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/376f8d69-b7c3-443d-9f54-4185d44bb6cf_isrForCrosstalkSources_1125917_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/376f8d69-b7c3-443d-9f54-4185d44bb6cf_isrForCrosstalkSources_1125917_55.stderr parsl.dataflow.dflow DEBUG: Task 2909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2909 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2909 has memoization hash e675d351561ec0dea990788deaa45a1f parsl.dataflow.memoization INFO: Task 2909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2909 try 0 launched on executor multi with executor id 2910 parsl.dataflow.dflow INFO: Standard output for task 2909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/1777e3e6-653e-49b5-8062-44ce3525eb1a_isrForCrosstalkSources_1125912_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/1777e3e6-653e-49b5-8062-44ce3525eb1a_isrForCrosstalkSources_1125912_6.stderr parsl.dataflow.dflow DEBUG: Task 2910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2910 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2910 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2910 has memoization hash 092bca851f76850b26b77954add5c55c parsl.dataflow.memoization INFO: Task 2910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2910 try 0 launched on executor multi with executor id 2911 parsl.dataflow.dflow INFO: Standard output for task 2910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/2224e72b-eb50-43b6-8c19-4bf52c00bb11_isrForCrosstalkSources_1125928_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/2224e72b-eb50-43b6-8c19-4bf52c00bb11_isrForCrosstalkSources_1125928_31.stderr parsl.dataflow.dflow DEBUG: Task 2911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2911 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2911 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2911 has memoization hash 0fbe0429c16d0ebb4b52b62714c613de parsl.dataflow.memoization INFO: Task 2911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2911 try 0 launched on executor multi with executor id 2912 parsl.dataflow.dflow INFO: Standard output for task 2911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/dc13f9bf-36a1-4df9-b111-5adec22948cd_isrForCrosstalkSources_1125908_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/dc13f9bf-36a1-4df9-b111-5adec22948cd_isrForCrosstalkSources_1125908_57.stderr parsl.dataflow.dflow DEBUG: Task 2912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2912 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2912 has memoization hash 93b537462d8602bfa45372c26d95e864 parsl.dataflow.memoization INFO: Task 2912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2912 try 0 launched on executor multi with executor id 2913 parsl.dataflow.dflow INFO: Standard output for task 2912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/d504f3fe-c496-4e5a-aa1d-369ce91be5c8_isrForCrosstalkSources_1125930_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/d504f3fe-c496-4e5a-aa1d-369ce91be5c8_isrForCrosstalkSources_1125930_12.stderr parsl.dataflow.dflow DEBUG: Task 2913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2913 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2913 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2913 has memoization hash 14e2b808e05dfac0b7e1353a1b1d3506 parsl.dataflow.memoization INFO: Task 2913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2913 try 0 launched on executor multi with executor id 2914 parsl.dataflow.dflow INFO: Standard output for task 2913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/06bb6c91-81a0-46da-b448-8c0c4cb5a1fd_isrForCrosstalkSources_1125934_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/06bb6c91-81a0-46da-b448-8c0c4cb5a1fd_isrForCrosstalkSources_1125934_58.stderr parsl.dataflow.dflow DEBUG: Task 2914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2914 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2914 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2914 has memoization hash 8da277c971246380fd5a0f6fcd03b87f parsl.dataflow.memoization INFO: Task 2914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2914 try 0 launched on executor multi with executor id 2915 parsl.dataflow.dflow INFO: Standard output for task 2914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/0f43f223-4859-4b84-8752-5bbf5c5bdcad_isrForCrosstalkSources_1125947_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/0f43f223-4859-4b84-8752-5bbf5c5bdcad_isrForCrosstalkSources_1125947_34.stderr parsl.dataflow.dflow DEBUG: Task 2915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2915 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2915 has memoization hash 97210851beec8015dbfde8b8b8755a1e parsl.dataflow.memoization INFO: Task 2915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2915 try 0 launched on executor multi with executor id 2916 parsl.dataflow.dflow INFO: Standard output for task 2915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/49209465-c317-4587-9405-3565e3e06c87_isrForCrosstalkSources_1125911_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/49209465-c317-4587-9405-3565e3e06c87_isrForCrosstalkSources_1125911_38.stderr parsl.dataflow.dflow DEBUG: Task 2916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2916 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2916 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2916 has memoization hash aa63dd6df665b7223daa1b4286f0d389 parsl.dataflow.memoization INFO: Task 2916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2916 try 0 launched on executor multi with executor id 2917 parsl.dataflow.dflow INFO: Standard output for task 2916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/dee6e409-3b86-44fa-a7a5-3940f58250d0_isrForCrosstalkSources_1125894_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/dee6e409-3b86-44fa-a7a5-3940f58250d0_isrForCrosstalkSources_1125894_44.stderr parsl.dataflow.dflow DEBUG: Task 2917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies 2024-03-07 15:07:03 proc_lsst.multi:170 [INFO] [multi] job_id 16896430 parsl.dataflow.dflow INFO: Task 2917 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2917 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2917 has memoization hash 29b293a4e675ae95a6a7d11baaefa0a1 parsl.dataflow.memoization INFO: Task 2917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) proc_lsst.multi INFO: [multi] job_id 16896430 2024-03-07 15:07:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 15:07:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:03 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896430 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896430 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 16896430 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-07 15:07:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-07 15:07:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 15:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2917 try 0 launched on executor multi with executor id 2918 parsl.dataflow.dflow INFO: Standard output for task 2917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/bf5ddc03-1ae9-4fa4-ae5c-629d175d5b8d_isrForCrosstalkSources_1125914_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/bf5ddc03-1ae9-4fa4-ae5c-629d175d5b8d_isrForCrosstalkSources_1125914_50.stderr parsl.dataflow.dflow DEBUG: Task 2918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2918 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2918 has memoization hash f20e0cefc85c1a53b73d2478961d81cd parsl.dataflow.memoization INFO: Task 2918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a396dad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2918 try 0 launched on executor multi with executor id 2919 parsl.dataflow.dflow INFO: Standard output for task 2918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/688e6f1e-6596-41bd-ab73-efa8e5338add_isrForCrosstalkSources_1125909_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/688e6f1e-6596-41bd-ab73-efa8e5338add_isrForCrosstalkSources_1125909_5.stderr parsl.dataflow.dflow DEBUG: Task 2919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2919 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2919 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2919 has memoization hash 26611ebd1f8e88d283f49ca6ca971816 parsl.dataflow.memoization INFO: Task 2919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2919 try 0 launched on executor multi with executor id 2920 parsl.dataflow.dflow INFO: Standard output for task 2919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/982fb279-5b31-4307-9610-44539aca0085_isrForCrosstalkSources_1125901_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/982fb279-5b31-4307-9610-44539aca0085_isrForCrosstalkSources_1125901_31.stderr parsl.dataflow.dflow DEBUG: Task 2920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2920 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2920 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2920 has memoization hash 6c2961f685fd05405bd0d9ca49920d7a parsl.dataflow.memoization INFO: Task 2920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2920 try 0 launched on executor multi with executor id 2921 parsl.dataflow.dflow INFO: Standard output for task 2920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/e8bad1f4-61ce-4824-8b6c-1471712b378b_isrForCrosstalkSources_1125917_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/e8bad1f4-61ce-4824-8b6c-1471712b378b_isrForCrosstalkSources_1125917_10.stderr parsl.dataflow.dflow DEBUG: Task 2921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2921 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2921 has memoization hash 29684d2c8b2ba60b80c46ebc7f0481c9 parsl.dataflow.memoization INFO: Task 2921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2921 try 0 launched on executor multi with executor id 2922 parsl.dataflow.dflow INFO: Standard output for task 2921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/067e8cb4-51f9-4a56-9420-fb6ccc520636_isrForCrosstalkSources_1125907_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/067e8cb4-51f9-4a56-9420-fb6ccc520636_isrForCrosstalkSources_1125907_26.stderr parsl.dataflow.dflow DEBUG: Task 2922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2922 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2922 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2922 has memoization hash 3f439b95bbeee595582df7f4f3985f74 parsl.dataflow.memoization INFO: Task 2922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2922 try 0 launched on executor multi with executor id 2923 parsl.dataflow.dflow INFO: Standard output for task 2922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/bda1bcc6-588e-4111-a751-fe0ec6ca9034_isrForCrosstalkSources_1125898_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/bda1bcc6-588e-4111-a751-fe0ec6ca9034_isrForCrosstalkSources_1125898_13.stderr parsl.dataflow.dflow DEBUG: Task 2923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2923 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2923 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2923 has memoization hash 574cf1fd6e35ca0b2e7fb81e3a5d52db parsl.dataflow.memoization INFO: Task 2923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2923 try 0 launched on executor multi with executor id 2924 parsl.dataflow.dflow INFO: Standard output for task 2923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/9fc38d0b-a1a6-4fb8-9744-45a97e4e1b04_isrForCrosstalkSources_1125930_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/9fc38d0b-a1a6-4fb8-9744-45a97e4e1b04_isrForCrosstalkSources_1125930_4.stderr parsl.dataflow.dflow DEBUG: Task 2924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2924 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2924 has memoization hash 29cdb1775842a155f2acdd0f9c870e40 parsl.dataflow.memoization INFO: Task 2924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2924 try 0 launched on executor multi with executor id 2925 parsl.dataflow.dflow INFO: Standard output for task 2924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8abf87ac-7791-45f0-b7be-308918414948_isrForCrosstalkSources_1125891_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8abf87ac-7791-45f0-b7be-308918414948_isrForCrosstalkSources_1125891_37.stderr parsl.dataflow.dflow DEBUG: Task 2925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2925 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2925 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2925 has memoization hash ed23fb322c8fe6ad5bac1860f8a44912 parsl.dataflow.memoization INFO: Task 2925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2925 try 0 launched on executor multi with executor id 2926 parsl.dataflow.dflow INFO: Standard output for task 2925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/9788cf66-1efb-4588-920c-2eaae3bc8c8a_isrForCrosstalkSources_1125925_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/9788cf66-1efb-4588-920c-2eaae3bc8c8a_isrForCrosstalkSources_1125925_4.stderr parsl.dataflow.dflow DEBUG: Task 2926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2926 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2926 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2926 has memoization hash 40b00474f136f05ca55d4e1b50b48bd1 parsl.dataflow.memoization INFO: Task 2926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2926 try 0 launched on executor multi with executor id 2927 parsl.dataflow.dflow INFO: Standard output for task 2926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/1b564fad-9e35-4ebe-bbbe-6b694141c2f9_isrForCrosstalkSources_1125916_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/1b564fad-9e35-4ebe-bbbe-6b694141c2f9_isrForCrosstalkSources_1125916_60.stderr parsl.dataflow.dflow DEBUG: Task 2927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2927 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2927 has memoization hash 905f051ed16340e3017fbeb7f060069e parsl.dataflow.memoization INFO: Task 2927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2927 try 0 launched on executor multi with executor id 2928 parsl.dataflow.dflow INFO: Standard output for task 2927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f3698660-d36f-45d7-ba21-ff92b47309fa_isrForCrosstalkSources_1125945_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f3698660-d36f-45d7-ba21-ff92b47309fa_isrForCrosstalkSources_1125945_10.stderr parsl.dataflow.dflow DEBUG: Task 2928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2928 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2928 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2928 has memoization hash 39861f2baf6686c8483807e3aa49f554 parsl.dataflow.memoization INFO: Task 2928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2928 try 0 launched on executor multi with executor id 2929 parsl.dataflow.dflow INFO: Standard output for task 2928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/674c7253-d180-4261-90e0-4af7bda065d1_isrForCrosstalkSources_1125912_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/674c7253-d180-4261-90e0-4af7bda065d1_isrForCrosstalkSources_1125912_12.stderr parsl.dataflow.dflow DEBUG: Task 2929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2929 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2929 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2929 has memoization hash 9fe0e5329dac5ffd18050e581b72788f parsl.dataflow.memoization INFO: Task 2929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2929 try 0 launched on executor multi with executor id 2930 parsl.dataflow.dflow INFO: Standard output for task 2929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/419bb142-85bf-4343-98ad-921145761ece_isrForCrosstalkSources_1125900_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/419bb142-85bf-4343-98ad-921145761ece_isrForCrosstalkSources_1125900_37.stderr parsl.dataflow.dflow DEBUG: Task 2930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2930 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2930 has memoization hash e31d2885178bf63f50a61c6548bb6d01 parsl.dataflow.memoization INFO: Task 2930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2930 try 0 launched on executor multi with executor id 2931 parsl.dataflow.dflow INFO: Standard output for task 2930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/091a19c4-5b6d-4fbe-98de-2305d1be7b0a_isrForCrosstalkSources_1125911_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/091a19c4-5b6d-4fbe-98de-2305d1be7b0a_isrForCrosstalkSources_1125911_17.stderr parsl.dataflow.dflow DEBUG: Task 2931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2931 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2931 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2931 has memoization hash a2a0ce9a79f8f1fa7ab62a886c4feb8c parsl.dataflow.memoization INFO: Task 2931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2931 try 0 launched on executor multi with executor id 2932 parsl.dataflow.dflow INFO: Standard output for task 2931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f97ca5f1-f1c2-4a7b-af7a-6e1c4e5984f0_isrForCrosstalkSources_1125897_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f97ca5f1-f1c2-4a7b-af7a-6e1c4e5984f0_isrForCrosstalkSources_1125897_53.stderr parsl.dataflow.dflow DEBUG: Task 2932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2932 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2932 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2932 has memoization hash 776824152a412fdc84cfe0eff615b714 parsl.dataflow.memoization INFO: Task 2932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2932 try 0 launched on executor multi with executor id 2933 parsl.dataflow.dflow INFO: Standard output for task 2932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/47f065b4-0862-4952-b6fe-85769fbc4e66_isrForCrosstalkSources_1125894_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/47f065b4-0862-4952-b6fe-85769fbc4e66_isrForCrosstalkSources_1125894_1.stderr parsl.dataflow.dflow DEBUG: Task 2933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2933 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2933 has memoization hash f0f5384300578211b791758a272c8720 parsl.dataflow.memoization INFO: Task 2933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2933 try 0 launched on executor multi with executor id 2934 parsl.dataflow.dflow INFO: Standard output for task 2933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/6169340f-3d13-48a1-9246-d7992e1253ea_isrForCrosstalkSources_1125907_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/6169340f-3d13-48a1-9246-d7992e1253ea_isrForCrosstalkSources_1125907_48.stderr parsl.dataflow.dflow DEBUG: Task 2934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2934 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2934 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2934 has memoization hash 7b0011012fc21478fae075ef2df9932e parsl.dataflow.memoization INFO: Task 2934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2934 try 0 launched on executor multi with executor id 2935 parsl.dataflow.dflow INFO: Standard output for task 2934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/2f432a00-4f8f-4f44-840f-a08aefc3fede_isrForCrosstalkSources_1125922_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/2f432a00-4f8f-4f44-840f-a08aefc3fede_isrForCrosstalkSources_1125922_48.stderr parsl.dataflow.dflow DEBUG: Task 2935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2935 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2935 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2935 has memoization hash 205d407d0384ad4272e68ea17aa85067 parsl.dataflow.memoization INFO: Task 2935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2935 try 0 launched on executor multi with executor id 2936 parsl.dataflow.dflow INFO: Standard output for task 2935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/ee1c7d7e-2df7-46dc-839e-9c57ef9ca1f0_isrForCrosstalkSources_1125923_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/ee1c7d7e-2df7-46dc-839e-9c57ef9ca1f0_isrForCrosstalkSources_1125923_47.stderr parsl.dataflow.dflow DEBUG: Task 2936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2936 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2936 has memoization hash 6330e3081dd836a84fc5631f640b032b parsl.dataflow.memoization INFO: Task 2936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2936 try 0 launched on executor multi with executor id 2937 parsl.dataflow.dflow INFO: Standard output for task 2936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/10d76b01-9b99-4ed0-9616-2d647d878289_isrForCrosstalkSources_1125907_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/10d76b01-9b99-4ed0-9616-2d647d878289_isrForCrosstalkSources_1125907_45.stderr parsl.dataflow.dflow DEBUG: Task 2937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2937 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2937 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2937 has memoization hash 31e0795f57b37732b4753afb251a61b0 parsl.dataflow.memoization INFO: Task 2937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2937 try 0 launched on executor multi with executor id 2938 parsl.dataflow.dflow INFO: Standard output for task 2937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/41b12a68-18e7-4b66-9e50-284c44177dcd_isrForCrosstalkSources_1125928_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/41b12a68-18e7-4b66-9e50-284c44177dcd_isrForCrosstalkSources_1125928_42.stderr parsl.dataflow.dflow DEBUG: Task 2938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2938 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2938 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2938 has memoization hash abc016debc744e91e5e1e07639b95cfa parsl.dataflow.memoization INFO: Task 2938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2938 try 0 launched on executor multi with executor id 2939 parsl.dataflow.dflow INFO: Standard output for task 2938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/c9f5923d-d067-4e8b-a0dc-62e02402c483_isrForCrosstalkSources_1125901_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/c9f5923d-d067-4e8b-a0dc-62e02402c483_isrForCrosstalkSources_1125901_59.stderr parsl.dataflow.dflow DEBUG: Task 2939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2939 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2939 has memoization hash 5b484810a103793f8a2acdb61341fd32 parsl.dataflow.memoization INFO: Task 2939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2939 try 0 launched on executor multi with executor id 2940 parsl.dataflow.dflow INFO: Standard output for task 2939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/5b913757-d4ea-49b2-99e2-eb5fdc646b6c_isrForCrosstalkSources_1125927_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/5b913757-d4ea-49b2-99e2-eb5fdc646b6c_isrForCrosstalkSources_1125927_53.stderr parsl.dataflow.dflow DEBUG: Task 2940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2940 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2940 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2940 has memoization hash 9171719a99fb89b7c5aa9422a098828a parsl.dataflow.memoization INFO: Task 2940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2940 try 0 launched on executor multi with executor id 2941 parsl.dataflow.dflow INFO: Standard output for task 2940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/acbcd238-3ef1-4fca-9fbf-daff73eb3a98_isrForCrosstalkSources_1125900_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/acbcd238-3ef1-4fca-9fbf-daff73eb3a98_isrForCrosstalkSources_1125900_14.stderr parsl.dataflow.dflow DEBUG: Task 2941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2941 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2941 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2941 has memoization hash c02f1c5edce5846a4de9561e4ed1481c parsl.dataflow.memoization INFO: Task 2941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2941 try 0 launched on executor multi with executor id 2942 parsl.dataflow.dflow INFO: Standard output for task 2941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/6aae6415-e57a-4dc2-a467-35f5a58c8aeb_isrForCrosstalkSources_1125926_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/6aae6415-e57a-4dc2-a467-35f5a58c8aeb_isrForCrosstalkSources_1125926_28.stderr parsl.dataflow.dflow DEBUG: Task 2942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2942 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2942 has memoization hash 5c2144597d9d283e0708fa0595647dc3 parsl.dataflow.memoization INFO: Task 2942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2942 try 0 launched on executor multi with executor id 2943 parsl.dataflow.dflow INFO: Standard output for task 2942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/9641f867-2960-42d9-8897-fb2288d7a22c_isrForCrosstalkSources_1125913_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/9641f867-2960-42d9-8897-fb2288d7a22c_isrForCrosstalkSources_1125913_22.stderr parsl.dataflow.dflow DEBUG: Task 2943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2943 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2943 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2943 has memoization hash 864fc191967d4204c4822ee34238b2a9 parsl.dataflow.memoization INFO: Task 2943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2943 try 0 launched on executor multi with executor id 2944 parsl.dataflow.dflow INFO: Standard output for task 2943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/1e41ce34-936d-41d1-94de-14968589ca8b_isrForCrosstalkSources_1125889_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/1e41ce34-936d-41d1-94de-14968589ca8b_isrForCrosstalkSources_1125889_42.stderr parsl.dataflow.dflow DEBUG: Task 2944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2944 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2944 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2944 has memoization hash b54096110f8626ddf4fd153e05287806 parsl.dataflow.memoization INFO: Task 2944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2944 try 0 launched on executor multi with executor id 2945 parsl.dataflow.dflow INFO: Standard output for task 2944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/c45a72bf-2976-4c4b-bee9-7ceafa1e5697_isrForCrosstalkSources_1125926_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/c45a72bf-2976-4c4b-bee9-7ceafa1e5697_isrForCrosstalkSources_1125926_48.stderr parsl.dataflow.dflow DEBUG: Task 2945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2945 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2945 has memoization hash b50d27cc950c91e39860f15b83ae4599 parsl.dataflow.memoization INFO: Task 2945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2945 try 0 launched on executor multi with executor id 2946 parsl.dataflow.dflow INFO: Standard output for task 2945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/feb3c2fc-1d39-49f2-a994-b26806ef4953_isrForCrosstalkSources_1125908_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/feb3c2fc-1d39-49f2-a994-b26806ef4953_isrForCrosstalkSources_1125908_1.stderr parsl.dataflow.dflow DEBUG: Task 2946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2946 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2946 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2946 has memoization hash 985de1766049d1c83939f3e081e17b2e parsl.dataflow.memoization INFO: Task 2946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2946 try 0 launched on executor multi with executor id 2947 parsl.dataflow.dflow INFO: Standard output for task 2946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/deacee88-e1ed-4f34-b9f1-8ddeb1201f4b_isrForCrosstalkSources_1125896_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/deacee88-e1ed-4f34-b9f1-8ddeb1201f4b_isrForCrosstalkSources_1125896_34.stderr parsl.dataflow.dflow DEBUG: Task 2947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2947 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2947 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2947 has memoization hash 1808fbad27dcb952e6282ca9ec257e75 parsl.dataflow.memoization INFO: Task 2947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2947 try 0 launched on executor multi with executor id 2948 parsl.dataflow.dflow INFO: Standard output for task 2947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/34d1c081-8007-47d1-9762-2a4ab7e01a92_isrForCrosstalkSources_1125939_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/34d1c081-8007-47d1-9762-2a4ab7e01a92_isrForCrosstalkSources_1125939_29.stderr parsl.dataflow.dflow DEBUG: Task 2948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2948 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2948 has memoization hash a74c062c11f524a3c4458b06fe558fba parsl.dataflow.memoization INFO: Task 2948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2948 try 0 launched on executor multi with executor id 2949 parsl.dataflow.dflow INFO: Standard output for task 2948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/d44992c2-5e0a-4155-b0de-06a7edf90e51_isrForCrosstalkSources_1125929_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/d44992c2-5e0a-4155-b0de-06a7edf90e51_isrForCrosstalkSources_1125929_47.stderr parsl.dataflow.dflow DEBUG: Task 2949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2949 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2949 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2949 has memoization hash 5ef416fb85062003dec4b1d8d8005fe6 parsl.dataflow.memoization INFO: Task 2949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2949 try 0 launched on executor multi with executor id 2950 parsl.dataflow.dflow INFO: Standard output for task 2949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a7ba0ffa-591d-4db6-a0ea-b90cd2f34b55_isrForCrosstalkSources_1125937_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a7ba0ffa-591d-4db6-a0ea-b90cd2f34b55_isrForCrosstalkSources_1125937_26.stderr parsl.dataflow.dflow DEBUG: Task 2950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2950 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2950 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2950 has memoization hash a1b6292b41b28e6be7f68b35401fb188 parsl.dataflow.memoization INFO: Task 2950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2950 try 0 launched on executor multi with executor id 2951 parsl.dataflow.dflow INFO: Standard output for task 2950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/e6435a85-4e91-47eb-a796-2446245cb99d_isrForCrosstalkSources_1125923_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/e6435a85-4e91-47eb-a796-2446245cb99d_isrForCrosstalkSources_1125923_7.stderr parsl.dataflow.dflow DEBUG: Task 2951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2951 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2951 has memoization hash 7d8bffe97d92f5bbd5100ddf2e6f4b16 parsl.dataflow.memoization INFO: Task 2951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2951 try 0 launched on executor multi with executor id 2952 parsl.dataflow.dflow INFO: Standard output for task 2951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/67595fff-92fe-419a-aecb-34e71d580c47_isrForCrosstalkSources_1125934_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/67595fff-92fe-419a-aecb-34e71d580c47_isrForCrosstalkSources_1125934_29.stderr parsl.dataflow.dflow DEBUG: Task 2952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2952 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2952 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2952 has memoization hash 9dd977e3d7c7fb31b44914f1aa34d0bd parsl.dataflow.memoization INFO: Task 2952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2952 try 0 launched on executor multi with executor id 2953 parsl.dataflow.dflow INFO: Standard output for task 2952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/898940c9-90cb-43a7-9d63-d9b4df39fbda_isrForCrosstalkSources_1125920_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/898940c9-90cb-43a7-9d63-d9b4df39fbda_isrForCrosstalkSources_1125920_51.stderr parsl.dataflow.dflow DEBUG: Task 2953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2953 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2953 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2953 has memoization hash f631a4600b066fa710b51ddfe859eac3 parsl.dataflow.memoization INFO: Task 2953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2953 try 0 launched on executor multi with executor id 2954 parsl.dataflow.dflow INFO: Standard output for task 2953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/6e453841-3110-48b6-b8a9-ea608251336a_isrForCrosstalkSources_1125911_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/6e453841-3110-48b6-b8a9-ea608251336a_isrForCrosstalkSources_1125911_54.stderr parsl.dataflow.dflow DEBUG: Task 2954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2954 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2954 has memoization hash 685a407cd82b8a137832b3ae363f1bf1 parsl.dataflow.memoization INFO: Task 2954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2954 try 0 launched on executor multi with executor id 2955 parsl.dataflow.dflow INFO: Standard output for task 2954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/6f0a56f0-46d8-4ab4-8c13-b0d7d5562380_isrForCrosstalkSources_1125935_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/6f0a56f0-46d8-4ab4-8c13-b0d7d5562380_isrForCrosstalkSources_1125935_29.stderr parsl.dataflow.dflow DEBUG: Task 2955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2955 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2955 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2955 has memoization hash dbbb2583b6c13a2e92595a35fe070d93 parsl.dataflow.memoization INFO: Task 2955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2955 try 0 launched on executor multi with executor id 2956 parsl.dataflow.dflow INFO: Standard output for task 2955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/78e8f2e8-87f6-43fe-bf48-7fd3663e992e_isrForCrosstalkSources_1125930_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/78e8f2e8-87f6-43fe-bf48-7fd3663e992e_isrForCrosstalkSources_1125930_57.stderr parsl.dataflow.dflow DEBUG: Task 2956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2956 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2956 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2956 has memoization hash 37928195b1fad719d207fde296aebe45 parsl.dataflow.memoization INFO: Task 2956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2956 try 0 launched on executor multi with executor id 2957 parsl.dataflow.dflow INFO: Standard output for task 2956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/1aed101f-cd5e-450d-93fe-3d1ddc53b0d1_isrForCrosstalkSources_1125907_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/1aed101f-cd5e-450d-93fe-3d1ddc53b0d1_isrForCrosstalkSources_1125907_37.stderr parsl.dataflow.dflow DEBUG: Task 2957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2957 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2957 has memoization hash 7bcd2d75088430ff879acf2e2b4c966c parsl.dataflow.memoization INFO: Task 2957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2957 try 0 launched on executor multi with executor id 2958 parsl.dataflow.dflow INFO: Standard output for task 2957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/cee2b05e-d427-47c8-9f10-fdc48d4a7acf_isrForCrosstalkSources_1125914_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/cee2b05e-d427-47c8-9f10-fdc48d4a7acf_isrForCrosstalkSources_1125914_31.stderr parsl.dataflow.dflow DEBUG: Task 2958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2958 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2958 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2958 has memoization hash af3c550c91cec7c3f7e355d14b01fe50 parsl.dataflow.memoization INFO: Task 2958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:03 proc_lsst.multi:170 [INFO] [multi] job_id 16896431 proc_lsst.multi INFO: [multi] job_id 16896431 2024-03-07 15:07:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 15:07:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:03 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896431 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896431 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 16896431 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-07 15:07:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-07 15:07:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 15:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2958 try 0 launched on executor multi with executor id 2959 parsl.dataflow.dflow INFO: Standard output for task 2958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/6a10bad2-fbaf-4d92-a61b-000d024e8c98_isrForCrosstalkSources_1125917_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/6a10bad2-fbaf-4d92-a61b-000d024e8c98_isrForCrosstalkSources_1125917_27.stderr parsl.dataflow.dflow DEBUG: Task 2959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2959 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2959 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2959 has memoization hash 3314220312f1b7a1277b3869b0a19a65 parsl.dataflow.memoization INFO: Task 2959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2959 try 0 launched on executor multi with executor id 2960 parsl.dataflow.dflow INFO: Standard output for task 2959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b0c15585-3673-4e1e-8516-221a1b3c0829_isrForCrosstalkSources_1125934_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b0c15585-3673-4e1e-8516-221a1b3c0829_isrForCrosstalkSources_1125934_17.stderr parsl.dataflow.dflow DEBUG: Task 2960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2960 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2960 has memoization hash 2529f2348348fd7e569b14c97071573a parsl.dataflow.memoization INFO: Task 2960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2960 try 0 launched on executor multi with executor id 2961 parsl.dataflow.dflow INFO: Standard output for task 2960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/87ec1d2a-6c39-4c64-bce3-a5b5f9aaf601_isrForCrosstalkSources_1125935_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/87ec1d2a-6c39-4c64-bce3-a5b5f9aaf601_isrForCrosstalkSources_1125935_59.stderr parsl.dataflow.dflow DEBUG: Task 2961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2961 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2961 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2961 has memoization hash a78c54a1d0121ea848a6bf305109e5ed parsl.dataflow.memoization INFO: Task 2961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 2961 try 0 launched on executor multi with executor id 2962 parsl.dataflow.dflow INFO: Standard output for task 2961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/6f7fcab6-c24b-4ca0-b260-c6997557542b_isrForCrosstalkSources_1125922_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/6f7fcab6-c24b-4ca0-b260-c6997557542b_isrForCrosstalkSources_1125922_45.stderr parsl.dataflow.dflow DEBUG: Task 2962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2962 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2962 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2962 has memoization hash 2972095ecdca9fe50d2a5847ba1abd98 parsl.dataflow.memoization INFO: Task 2962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2962 try 0 launched on executor multi with executor id 2963 parsl.dataflow.dflow INFO: Standard output for task 2962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/7a348888-7527-4810-a04f-c93c45ee78aa_isrForCrosstalkSources_1125936_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/7a348888-7527-4810-a04f-c93c45ee78aa_isrForCrosstalkSources_1125936_31.stderr parsl.dataflow.dflow DEBUG: Task 2963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2963 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2963 has memoization hash 2301d5f3939a49d0dd36a5e384ed1410 parsl.dataflow.memoization INFO: Task 2963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2963 try 0 launched on executor multi with executor id 2964 parsl.dataflow.dflow INFO: Standard output for task 2963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/96f0bd27-2ccf-4db4-bdfa-275b99e7cb83_isrForCrosstalkSources_1125927_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/96f0bd27-2ccf-4db4-bdfa-275b99e7cb83_isrForCrosstalkSources_1125927_41.stderr parsl.dataflow.dflow DEBUG: Task 2964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2964 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2964 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2964 has memoization hash 3bf4915fd27ca1f51198a08cb52438cd parsl.dataflow.memoization INFO: Task 2964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2964 try 0 launched on executor multi with executor id 2965 parsl.dataflow.dflow INFO: Standard output for task 2964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/fa0ae72a-7391-460c-b50d-1fd1409a08ac_isrForCrosstalkSources_1125919_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/fa0ae72a-7391-460c-b50d-1fd1409a08ac_isrForCrosstalkSources_1125919_40.stderr parsl.dataflow.dflow DEBUG: Task 2965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2965 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2965 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2965 has memoization hash f50724b7f9fe267668ceb850f2b2602c parsl.dataflow.memoization INFO: Task 2965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2965 try 0 launched on executor multi with executor id 2966 parsl.dataflow.dflow INFO: Standard output for task 2965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/a96dad51-66e3-4094-819b-694de0ffb483_isrForCrosstalkSources_1125913_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/a96dad51-66e3-4094-819b-694de0ffb483_isrForCrosstalkSources_1125913_24.stderr parsl.dataflow.dflow DEBUG: Task 2966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2966 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2966 has memoization hash 6464ec0661b3f011fb95e87331b08d3c parsl.dataflow.memoization INFO: Task 2966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2966 try 0 launched on executor multi with executor id 2967 parsl.dataflow.dflow INFO: Standard output for task 2966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/022a1b2f-d294-4449-b1de-160bd8d55706_isrForCrosstalkSources_1125942_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/022a1b2f-d294-4449-b1de-160bd8d55706_isrForCrosstalkSources_1125942_11.stderr parsl.dataflow.dflow DEBUG: Task 2967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2967 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2967 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2967 has memoization hash 8804dabd7d7d246cde8a6267fe9b0220 parsl.dataflow.memoization INFO: Task 2967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2967 try 0 launched on executor multi with executor id 2968 parsl.dataflow.dflow INFO: Standard output for task 2967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/7a5f46c7-11a8-42aa-8be8-15d7c0bd43e4_isrForCrosstalkSources_1125912_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/7a5f46c7-11a8-42aa-8be8-15d7c0bd43e4_isrForCrosstalkSources_1125912_20.stderr parsl.dataflow.dflow DEBUG: Task 2968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2968 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2968 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2968 has memoization hash e26e6750e5afc857cac9a1faf071c0fd parsl.dataflow.memoization INFO: Task 2968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393632e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2968 try 0 launched on executor multi with executor id 2969 parsl.dataflow.dflow INFO: Standard output for task 2968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/6616945f-f743-4df5-969c-deb674a2a990_isrForCrosstalkSources_1125947_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/6616945f-f743-4df5-969c-deb674a2a990_isrForCrosstalkSources_1125947_20.stderr parsl.dataflow.dflow DEBUG: Task 2969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2969 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2969 has memoization hash 18bb864b64b5eec8bad4f787c2a0ea76 parsl.dataflow.memoization INFO: Task 2969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2969 try 0 launched on executor multi with executor id 2970 parsl.dataflow.dflow INFO: Standard output for task 2969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f60651db-6714-4b39-ba8f-4728c3f2ff4f_isrForCrosstalkSources_1125910_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f60651db-6714-4b39-ba8f-4728c3f2ff4f_isrForCrosstalkSources_1125910_27.stderr parsl.dataflow.dflow DEBUG: Task 2970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2970 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2970 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2970 has memoization hash 1da4cf9c2467300755f3635de0803e8e parsl.dataflow.memoization INFO: Task 2970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2970 try 0 launched on executor multi with executor id 2971 parsl.dataflow.dflow INFO: Standard output for task 2970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/b46eb4e9-c53e-44be-9145-835b9588b339_isrForCrosstalkSources_1125918_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/b46eb4e9-c53e-44be-9145-835b9588b339_isrForCrosstalkSources_1125918_18.stderr parsl.dataflow.dflow DEBUG: Task 2971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2971 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2971 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2971 has memoization hash 3a456569229f0f826e4538d05fad0154 parsl.dataflow.memoization INFO: Task 2971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2971 try 0 launched on executor multi with executor id 2972 parsl.dataflow.dflow INFO: Standard output for task 2971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/70876277-f40f-492c-86af-1cc28f862fe2_isrForCrosstalkSources_1125921_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/70876277-f40f-492c-86af-1cc28f862fe2_isrForCrosstalkSources_1125921_25.stderr parsl.dataflow.dflow DEBUG: Task 2972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2972 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2972 has memoization hash 1aae5d46dfdc8a0d6dda4e70c706c8d6 parsl.dataflow.memoization INFO: Task 2972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393604a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2972 try 0 launched on executor multi with executor id 2973 parsl.dataflow.dflow INFO: Standard output for task 2972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/0661384d-fb2f-407b-ba94-c9dbd5e59588_isrForCrosstalkSources_1125934_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/0661384d-fb2f-407b-ba94-c9dbd5e59588_isrForCrosstalkSources_1125934_62.stderr parsl.dataflow.dflow DEBUG: Task 2973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2973 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2973 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2973 has memoization hash ac0ce21c28a85e9f25b8a7691ecc6a0f parsl.dataflow.memoization INFO: Task 2973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2973 try 0 launched on executor multi with executor id 2974 parsl.dataflow.dflow INFO: Standard output for task 2973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/f803717f-fdb0-4c7b-94d1-1b027948d54e_isrForCrosstalkSources_1125921_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/f803717f-fdb0-4c7b-94d1-1b027948d54e_isrForCrosstalkSources_1125921_58.stderr parsl.dataflow.dflow DEBUG: Task 2974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2974 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2974 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2974 has memoization hash 8aded4ffeea63a7d5454c3561e3667d7 parsl.dataflow.memoization INFO: Task 2974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393602c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2974 try 0 launched on executor multi with executor id 2975 parsl.dataflow.dflow INFO: Standard output for task 2974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/bd1929f9-a941-48b2-b69e-9fca18b433a7_isrForCrosstalkSources_1125906_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/bd1929f9-a941-48b2-b69e-9fca18b433a7_isrForCrosstalkSources_1125906_42.stderr parsl.dataflow.dflow DEBUG: Task 2975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2975 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2975 has memoization hash da76d45ac9294c89de2ef371b9875c46 parsl.dataflow.memoization INFO: Task 2975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2975 try 0 launched on executor multi with executor id 2976 parsl.dataflow.dflow INFO: Standard output for task 2975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/29066f96-219f-4e54-9087-6399f77c86be_isrForCrosstalkSources_1125931_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/29066f96-219f-4e54-9087-6399f77c86be_isrForCrosstalkSources_1125931_57.stderr parsl.dataflow.dflow DEBUG: Task 2976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2976 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2976 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2976 has memoization hash 9c97456a6e01236528f4370bfc382ec3 parsl.dataflow.memoization INFO: Task 2976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393605e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2976 try 0 launched on executor multi with executor id 2977 parsl.dataflow.dflow INFO: Standard output for task 2976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/ad78c7b0-6ce3-4e9f-b6fc-a10ac705a445_isrForCrosstalkSources_1125895_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/ad78c7b0-6ce3-4e9f-b6fc-a10ac705a445_isrForCrosstalkSources_1125895_36.stderr parsl.dataflow.dflow DEBUG: Task 2977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2977 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2977 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2977 has memoization hash d778bb9adfe23e1a2aae40b9debbc5ed parsl.dataflow.memoization INFO: Task 2977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393607c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2977 try 0 launched on executor multi with executor id 2978 parsl.dataflow.dflow INFO: Standard output for task 2977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a1a38c95-93c8-41a7-9c3d-a3d0fe660c0c_isrForCrosstalkSources_1125909_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a1a38c95-93c8-41a7-9c3d-a3d0fe660c0c_isrForCrosstalkSources_1125909_20.stderr parsl.dataflow.dflow DEBUG: Task 2978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2978 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2978 has memoization hash 8f66c32d9fd55cd5f78232e1f047b297 parsl.dataflow.memoization INFO: Task 2978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2978 try 0 launched on executor multi with executor id 2979 parsl.dataflow.dflow INFO: Standard output for task 2978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/11d2c61b-fd76-4241-bd0a-af9e01236766_isrForCrosstalkSources_1125947_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/11d2c61b-fd76-4241-bd0a-af9e01236766_isrForCrosstalkSources_1125947_48.stderr parsl.dataflow.dflow DEBUG: Task 2979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2979 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2979 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2979 has memoization hash b654a7955ccf2b3d44969fe2e3d257c7 parsl.dataflow.memoization INFO: Task 2979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2979 try 0 launched on executor multi with executor id 2980 parsl.dataflow.dflow INFO: Standard output for task 2979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/c4d456b4-89c8-4d8d-a5a7-4c9dd9836cde_isrForCrosstalkSources_1125930_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/c4d456b4-89c8-4d8d-a5a7-4c9dd9836cde_isrForCrosstalkSources_1125930_23.stderr parsl.dataflow.dflow DEBUG: Task 2980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2980 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2980 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2980 has memoization hash 8d42a683d57a1b3077c9443428d80a77 parsl.dataflow.memoization INFO: Task 2980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2980 try 0 launched on executor multi with executor id 2981 parsl.dataflow.dflow INFO: Standard output for task 2980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/20b2c8e6-e767-467c-8f88-5f2e734ea592_isrForCrosstalkSources_1125944_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/20b2c8e6-e767-467c-8f88-5f2e734ea592_isrForCrosstalkSources_1125944_62.stderr parsl.dataflow.dflow DEBUG: Task 2981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2981 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2981 has memoization hash 86e96f8715c7eb3173bc169496281e52 parsl.dataflow.memoization INFO: Task 2981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2981 try 0 launched on executor multi with executor id 2982 parsl.dataflow.dflow INFO: Standard output for task 2981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/5f0bb0d0-12ec-435f-8219-f763395ade6c_isrForCrosstalkSources_1125920_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/5f0bb0d0-12ec-435f-8219-f763395ade6c_isrForCrosstalkSources_1125920_35.stderr parsl.dataflow.dflow DEBUG: Task 2982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2982 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2982 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2982 has memoization hash 63bfd3df052f15d17a6f97f8b7f1fccf parsl.dataflow.memoization INFO: Task 2982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2982 try 0 launched on executor multi with executor id 2983 parsl.dataflow.dflow INFO: Standard output for task 2982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/c66802b8-2792-4cef-8f1c-b849787cb1d3_isrForCrosstalkSources_1125890_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/c66802b8-2792-4cef-8f1c-b849787cb1d3_isrForCrosstalkSources_1125890_17.stderr parsl.dataflow.dflow DEBUG: Task 2983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2983 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2983 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2983 has memoization hash c14029e844a453870e4826caf1704337 parsl.dataflow.memoization INFO: Task 2983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393609a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2983 try 0 launched on executor multi with executor id 2984 parsl.dataflow.dflow INFO: Standard output for task 2983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/36991a94-b1c7-403a-9fe1-6e5ae5d3d79f_isrForCrosstalkSources_1125929_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/36991a94-b1c7-403a-9fe1-6e5ae5d3d79f_isrForCrosstalkSources_1125929_41.stderr parsl.dataflow.dflow DEBUG: Task 2984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2984 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2984 has memoization hash b313a60ae8423339efa55e7a182cbb20 parsl.dataflow.memoization INFO: Task 2984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2984 try 0 launched on executor multi with executor id 2985 parsl.dataflow.dflow INFO: Standard output for task 2984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/37fea12b-7f0d-4463-8362-cb8a01d9b658_isrForCrosstalkSources_1125892_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/37fea12b-7f0d-4463-8362-cb8a01d9b658_isrForCrosstalkSources_1125892_47.stderr parsl.dataflow.dflow DEBUG: Task 2985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2985 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2985 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2985 has memoization hash e51ab9f086b954157e5169ebe1d467a2 parsl.dataflow.memoization INFO: Task 2985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2985 try 0 launched on executor multi with executor id 2986 parsl.dataflow.dflow INFO: Standard output for task 2985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/01ade9fa-bf85-47e7-b98c-33dd1fc7da45_isrForCrosstalkSources_1125935_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/01ade9fa-bf85-47e7-b98c-33dd1fc7da45_isrForCrosstalkSources_1125935_37.stderr parsl.dataflow.dflow DEBUG: Task 2986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2986 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2986 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2986 has memoization hash 6950a28cfff9e47db15a8e177366bc7a parsl.dataflow.memoization INFO: Task 2986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2986 try 0 launched on executor multi with executor id 2987 parsl.dataflow.dflow INFO: Standard output for task 2986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/57d9de57-cdd5-417a-acae-58b7acd76d97_isrForCrosstalkSources_1125938_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/57d9de57-cdd5-417a-acae-58b7acd76d97_isrForCrosstalkSources_1125938_47.stderr parsl.dataflow.dflow DEBUG: Task 2987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2987 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2987 has memoization hash 9f80ed3b1ce5a2e3594c6b0cbf2f38a2 parsl.dataflow.memoization INFO: Task 2987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2987 try 0 launched on executor multi with executor id 2988 parsl.dataflow.dflow INFO: Standard output for task 2987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/2b16b540-ee5c-4d3a-b11d-cd6c8c4d4fa5_isrForCrosstalkSources_1125920_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/2b16b540-ee5c-4d3a-b11d-cd6c8c4d4fa5_isrForCrosstalkSources_1125920_34.stderr parsl.dataflow.dflow DEBUG: Task 2988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2988 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2988 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2988 has memoization hash a56d457cae8ad4474b7c813e270e8c3a parsl.dataflow.memoization INFO: Task 2988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2988 try 0 launched on executor multi with executor id 2989 parsl.dataflow.dflow INFO: Standard output for task 2988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/e3c71953-d598-4e44-be4f-84dffeb22728_isrForCrosstalkSources_1125908_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/e3c71953-d598-4e44-be4f-84dffeb22728_isrForCrosstalkSources_1125908_22.stderr parsl.dataflow.dflow DEBUG: Task 2989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2989 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2989 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2989 has memoization hash 5909aaefb452d92a8ae23116cd9ef812 parsl.dataflow.memoization INFO: Task 2989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2989 try 0 launched on executor multi with executor id 2990 parsl.dataflow.dflow INFO: Standard output for task 2989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/fe5e7f6d-232a-4872-bda9-7969cece15b1_isrForCrosstalkSources_1125923_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/fe5e7f6d-232a-4872-bda9-7969cece15b1_isrForCrosstalkSources_1125923_23.stderr parsl.dataflow.dflow DEBUG: Task 2990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2990 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2990 has memoization hash 957c38cd1460e62e239e0894b89bade6 parsl.dataflow.memoization INFO: Task 2990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2990 try 0 launched on executor multi with executor id 2991 parsl.dataflow.dflow INFO: Standard output for task 2990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/39b39e23-4b1f-480a-a550-480a189a7923_isrForCrosstalkSources_1125925_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/39b39e23-4b1f-480a-a550-480a189a7923_isrForCrosstalkSources_1125925_59.stderr parsl.dataflow.dflow DEBUG: Task 2991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2991 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2991 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2991 has memoization hash 2123e63ec343e92ab71a4e8401881684 parsl.dataflow.memoization INFO: Task 2991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2991 try 0 launched on executor multi with executor id 2992 parsl.dataflow.dflow INFO: Standard output for task 2991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/f2b6a4bd-70d5-4586-ac0d-973c3ac94d16_isrForCrosstalkSources_1125900_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/f2b6a4bd-70d5-4586-ac0d-973c3ac94d16_isrForCrosstalkSources_1125900_9.stderr parsl.dataflow.dflow DEBUG: Task 2992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2992 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2992 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2992 has memoization hash d7da9cef402db5031b7252b15c2cdd0d parsl.dataflow.memoization INFO: Task 2992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2992 try 0 launched on executor multi with executor id 2993 parsl.dataflow.dflow INFO: Standard output for task 2992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/178d79aa-c80c-407c-b8c9-879d9f77f59e_isrForCrosstalkSources_1125900_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/178d79aa-c80c-407c-b8c9-879d9f77f59e_isrForCrosstalkSources_1125900_1.stderr parsl.dataflow.dflow DEBUG: Task 2993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2993 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2993 has memoization hash f59f5175f4f2610ec1bd7d628e1e5973 parsl.dataflow.memoization INFO: Task 2993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39360e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2993 try 0 launched on executor multi with executor id 2994 parsl.dataflow.dflow INFO: Standard output for task 2993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/cf771bea-a263-466f-8def-92fe741ec5d3_isrForCrosstalkSources_1125927_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/cf771bea-a263-466f-8def-92fe741ec5d3_isrForCrosstalkSources_1125927_19.stderr parsl.dataflow.dflow DEBUG: Task 2994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2994 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2994 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2994 has memoization hash a9b93629594a3d97b72ccd3868e81d89 parsl.dataflow.memoization INFO: Task 2994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2994 try 0 launched on executor multi with executor id 2995 parsl.dataflow.dflow INFO: Standard output for task 2994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/fb1cd9c7-21ff-47c2-9394-ba0284c01b32_isrForCrosstalkSources_1125893_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/fb1cd9c7-21ff-47c2-9394-ba0284c01b32_isrForCrosstalkSources_1125893_6.stderr parsl.dataflow.dflow DEBUG: Task 2995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2995 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2995 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2995 has memoization hash dd639630a9253f5e558bc6dcc38b37b8 parsl.dataflow.memoization INFO: Task 2995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393611c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2995 try 0 launched on executor multi with executor id 2996 parsl.dataflow.dflow INFO: Standard output for task 2995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/1d1bb996-feb4-4bfc-b527-1d55236519a1_isrForCrosstalkSources_1125947_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/1d1bb996-feb4-4bfc-b527-1d55236519a1_isrForCrosstalkSources_1125947_24.stderr parsl.dataflow.dflow DEBUG: Task 2996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2996 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2996 has memoization hash efa566550946689e331a45c303eabe05 parsl.dataflow.memoization INFO: Task 2996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:03 proc_lsst.multi:170 [INFO] [multi] job_id 16896432 proc_lsst.multi INFO: [multi] job_id 16896432 2024-03-07 15:07:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 15:07:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:03 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896432 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896432 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 16896432 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-07 15:07:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-07 15:07:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 15:07:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 2996 try 0 launched on executor multi with executor id 2997 parsl.dataflow.dflow INFO: Standard output for task 2996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/fef92aa7-a9a9-4747-bc16-160b66f39f12_isrForCrosstalkSources_1125900_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/fef92aa7-a9a9-4747-bc16-160b66f39f12_isrForCrosstalkSources_1125900_21.stderr parsl.dataflow.dflow DEBUG: Task 2997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2997 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2997 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2997 has memoization hash ce101a91500228a910911ce37786e426 parsl.dataflow.memoization INFO: Task 2997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2997 try 0 launched on executor multi with executor id 2998 parsl.dataflow.dflow INFO: Standard output for task 2997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/b78e250a-6987-42de-a98a-19c4da0ce26e_isrForCrosstalkSources_1125941_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/b78e250a-6987-42de-a98a-19c4da0ce26e_isrForCrosstalkSources_1125941_33.stderr parsl.dataflow.dflow DEBUG: Task 2998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2998 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2998 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2998 has memoization hash b3014a59957d3a9631e2a68e79e49a6e parsl.dataflow.memoization INFO: Task 2998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2998 try 0 launched on executor multi with executor id 2999 parsl.dataflow.dflow INFO: Standard output for task 2998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/8bbdf2e8-36c8-4a24-9f01-bb2d3ed9484b_isrForCrosstalkSources_1125915_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/8bbdf2e8-36c8-4a24-9f01-bb2d3ed9484b_isrForCrosstalkSources_1125915_14.stderr parsl.dataflow.dflow DEBUG: Task 2999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2999 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2999 has memoization hash c990bbade57bf429c7952c167c23e1de parsl.dataflow.memoization INFO: Task 2999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393613a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2999 try 0 launched on executor multi with executor id 3000 parsl.dataflow.dflow INFO: Standard output for task 2999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/10f0567b-3e96-42a8-a725-f9226505844e_isrForCrosstalkSources_1125890_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/10f0567b-3e96-42a8-a725-f9226505844e_isrForCrosstalkSources_1125890_41.stderr parsl.dataflow.dflow DEBUG: Task 3000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3000 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3000 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3000 has memoization hash 832dd59e870f09f5e616d4dcb7dd7509 parsl.dataflow.memoization INFO: Task 3000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393634c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3000 try 0 launched on executor multi with executor id 3001 parsl.dataflow.dflow INFO: Standard output for task 3000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/8e9eb5d0-2142-4dba-9f7a-6a52c8caaba0_isrForCrosstalkSources_1125939_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/8e9eb5d0-2142-4dba-9f7a-6a52c8caaba0_isrForCrosstalkSources_1125939_38.stderr parsl.dataflow.dflow DEBUG: Task 3001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3001 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3001 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3001 has memoization hash 1a460eac21a37ebcb0766644ac5ca8ad parsl.dataflow.memoization INFO: Task 3001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3001 try 0 launched on executor multi with executor id 3002 parsl.dataflow.dflow INFO: Standard output for task 3001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/79658052-88bc-49a5-ac25-3b01c2908c4d_isrForCrosstalkSources_1125908_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/79658052-88bc-49a5-ac25-3b01c2908c4d_isrForCrosstalkSources_1125908_34.stderr parsl.dataflow.dflow DEBUG: Task 3002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3002 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3002 has memoization hash f5ed477afdc499031da1676e83de45c5 parsl.dataflow.memoization INFO: Task 3002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393614e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3002 try 0 launched on executor multi with executor id 3003 parsl.dataflow.dflow INFO: Standard output for task 3002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/3c976cc0-e270-4ce2-8a3d-eef44f61f278_isrForCrosstalkSources_1125903_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/3c976cc0-e270-4ce2-8a3d-eef44f61f278_isrForCrosstalkSources_1125903_58.stderr parsl.dataflow.dflow DEBUG: Task 3003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3003 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3003 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3003 has memoization hash 384634b3a823a7fbd5aa7537376f6ab0 parsl.dataflow.memoization INFO: Task 3003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393600e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3003 try 0 launched on executor multi with executor id 3004 parsl.dataflow.dflow INFO: Standard output for task 3003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1c054595-3852-4bba-b4cb-024047fcfbc2_isrForCrosstalkSources_1125900_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1c054595-3852-4bba-b4cb-024047fcfbc2_isrForCrosstalkSources_1125900_36.stderr parsl.dataflow.dflow DEBUG: Task 3004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3004 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3004 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3004 has memoization hash e9ddc25f594f1510a6fb63d5fbd34124 parsl.dataflow.memoization INFO: Task 3004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3004 try 0 launched on executor multi with executor id 3005 parsl.dataflow.dflow INFO: Standard output for task 3004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/887331fd-ead6-43cc-9eb8-204029fbdb18_isrForCrosstalkSources_1125918_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/887331fd-ead6-43cc-9eb8-204029fbdb18_isrForCrosstalkSources_1125918_12.stderr parsl.dataflow.dflow DEBUG: Task 3005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3005 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3005 has memoization hash 2488ae56e3d6133ff608a7acc495f107 parsl.dataflow.memoization INFO: Task 3005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3005 try 0 launched on executor multi with executor id 3006 parsl.dataflow.dflow INFO: Standard output for task 3005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/e7490eec-72de-4667-887b-ecb66001a62d_isrForCrosstalkSources_1125898_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/e7490eec-72de-4667-887b-ecb66001a62d_isrForCrosstalkSources_1125898_54.stderr parsl.dataflow.dflow DEBUG: Task 3006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3006 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3006 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3006 has memoization hash 5718760da7345ba81e86533d4fd4c26c parsl.dataflow.memoization INFO: Task 3006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3006 try 0 launched on executor multi with executor id 3007 parsl.dataflow.dflow INFO: Standard output for task 3006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/43073ba5-c24f-4bdf-8e07-99edcdc7faa0_isrForCrosstalkSources_1125896_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/43073ba5-c24f-4bdf-8e07-99edcdc7faa0_isrForCrosstalkSources_1125896_43.stderr parsl.dataflow.dflow DEBUG: Task 3007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3007 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3007 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3007 has memoization hash 9cf0b47646a72f42734b67da41f73ec3 parsl.dataflow.memoization INFO: Task 3007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3007 try 0 launched on executor multi with executor id 3008 parsl.dataflow.dflow INFO: Standard output for task 3007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/f628086b-f665-4719-bd0b-22f03b3d60a4_isrForCrosstalkSources_1125904_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/f628086b-f665-4719-bd0b-22f03b3d60a4_isrForCrosstalkSources_1125904_61.stderr parsl.dataflow.dflow DEBUG: Task 3008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3008 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3008 has memoization hash 84ef20897f5419a7c3e041ba3d77f6c9 parsl.dataflow.memoization INFO: Task 3008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3008 try 0 launched on executor multi with executor id 3009 parsl.dataflow.dflow INFO: Standard output for task 3008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/d1119e89-3f11-48de-9140-459dedfb128b_isrForCrosstalkSources_1125909_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/d1119e89-3f11-48de-9140-459dedfb128b_isrForCrosstalkSources_1125909_50.stderr parsl.dataflow.dflow DEBUG: Task 3009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3009 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3009 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3009 has memoization hash 103cbfbf926f292c60f2fc6d2a54b3a7 parsl.dataflow.memoization INFO: Task 3009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3009 try 0 launched on executor multi with executor id 3010 parsl.dataflow.dflow INFO: Standard output for task 3009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/18bdd824-3fca-43e8-9a8e-3a2a54cfba6a_isrForCrosstalkSources_1125908_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/18bdd824-3fca-43e8-9a8e-3a2a54cfba6a_isrForCrosstalkSources_1125908_2.stderr parsl.dataflow.dflow DEBUG: Task 3010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3010 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3010 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3010 has memoization hash cf5138e010b16cc33f91f270197b6a73 parsl.dataflow.memoization INFO: Task 3010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3010 try 0 launched on executor multi with executor id 3011 parsl.dataflow.dflow INFO: Standard output for task 3010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/0627355b-1334-4157-be8c-cdd67685aef8_isrForCrosstalkSources_1125903_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/0627355b-1334-4157-be8c-cdd67685aef8_isrForCrosstalkSources_1125903_60.stderr parsl.dataflow.dflow DEBUG: Task 3011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3011 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3011 has memoization hash 48fe04cfe280c62ed2d40c66658eabcd parsl.dataflow.memoization INFO: Task 3011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3011 try 0 launched on executor multi with executor id 3012 parsl.dataflow.dflow INFO: Standard output for task 3011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d48b3025-9f2f-45ac-b1f9-bcd726c3cc1e_isrForCrosstalkSources_1125911_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d48b3025-9f2f-45ac-b1f9-bcd726c3cc1e_isrForCrosstalkSources_1125911_47.stderr parsl.dataflow.dflow DEBUG: Task 3012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3012 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3012 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3012 has memoization hash 4096c9d5919f7393daa37e9889659322 parsl.dataflow.memoization INFO: Task 3012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3012 try 0 launched on executor multi with executor id 3013 parsl.dataflow.dflow INFO: Standard output for task 3012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/4156e10b-0984-4026-968b-b8093d03f874_isrForCrosstalkSources_1125892_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/4156e10b-0984-4026-968b-b8093d03f874_isrForCrosstalkSources_1125892_53.stderr parsl.dataflow.dflow DEBUG: Task 3013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3013 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3013 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3013 has memoization hash f4c5464c40122ed95be13e486b029acc parsl.dataflow.memoization INFO: Task 3013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3013 try 0 launched on executor multi with executor id 3014 parsl.dataflow.dflow INFO: Standard output for task 3013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/172f6b6f-cf3b-4040-aac7-04a4b47057b3_isrForCrosstalkSources_1125924_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/172f6b6f-cf3b-4040-aac7-04a4b47057b3_isrForCrosstalkSources_1125924_18.stderr parsl.dataflow.dflow DEBUG: Task 3014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3014 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3014 has memoization hash 0c092ef45fc2bdeb758f67f8b2f79344 parsl.dataflow.memoization INFO: Task 3014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3014 try 0 launched on executor multi with executor id 3015 parsl.dataflow.dflow INFO: Standard output for task 3014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/72385140-c694-4a61-88f9-2e04ebb7c697_isrForCrosstalkSources_1125928_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/72385140-c694-4a61-88f9-2e04ebb7c697_isrForCrosstalkSources_1125928_54.stderr parsl.dataflow.dflow DEBUG: Task 3015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3015 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3015 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3015 has memoization hash edfe436d632d57483aa702b9f06a42e4 parsl.dataflow.memoization INFO: Task 3015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393619e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3015 try 0 launched on executor multi with executor id 3016 parsl.dataflow.dflow INFO: Standard output for task 3015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/b82e9498-79bd-45cc-9706-d16bf60f8b2f_isrForCrosstalkSources_1125916_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/b82e9498-79bd-45cc-9706-d16bf60f8b2f_isrForCrosstalkSources_1125916_18.stderr parsl.dataflow.dflow DEBUG: Task 3016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3016 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3016 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3016 has memoization hash e29dcd465be226598341781b68373607 parsl.dataflow.memoization INFO: Task 3016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3016 try 0 launched on executor multi with executor id 3017 parsl.dataflow.dflow INFO: Standard output for task 3016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/031d099e-55ec-439e-bb04-269710bebccc_isrForCrosstalkSources_1125945_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/031d099e-55ec-439e-bb04-269710bebccc_isrForCrosstalkSources_1125945_61.stderr parsl.dataflow.dflow DEBUG: Task 3017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3017 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3017 has memoization hash b30fe3960f0059ac5adfbd8fbe9a23f2 parsl.dataflow.memoization INFO: Task 3017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3017 try 0 launched on executor multi with executor id 3018 parsl.dataflow.dflow INFO: Standard output for task 3017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/cbf19a4d-2f2d-4236-ab69-e4050cefc7dc_isrForCrosstalkSources_1125896_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/cbf19a4d-2f2d-4236-ab69-e4050cefc7dc_isrForCrosstalkSources_1125896_33.stderr parsl.dataflow.dflow DEBUG: Task 3018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3018 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3018 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3018 has memoization hash 1797a4db5236a7fb204980388547f7f5 parsl.dataflow.memoization INFO: Task 3018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3018 try 0 launched on executor multi with executor id 3019 parsl.dataflow.dflow INFO: Standard output for task 3018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/0157886d-693b-4a22-899d-4f0a644428b4_isrForCrosstalkSources_1125915_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/0157886d-693b-4a22-899d-4f0a644428b4_isrForCrosstalkSources_1125915_34.stderr parsl.dataflow.dflow DEBUG: Task 3019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3019 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3019 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3019 has memoization hash 7f62d03a096e872d87cfec204286f6f1 parsl.dataflow.memoization INFO: Task 3019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3019 try 0 launched on executor multi with executor id 3020 parsl.dataflow.dflow INFO: Standard output for task 3019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/9d0a2aa7-4d02-4fd7-9270-8d5c91d7fae8_isrForCrosstalkSources_1125888_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/9d0a2aa7-4d02-4fd7-9270-8d5c91d7fae8_isrForCrosstalkSources_1125888_46.stderr parsl.dataflow.dflow DEBUG: Task 3020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3020 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3020 has memoization hash 566bfe5733ffee16ddf3037342121529 parsl.dataflow.memoization INFO: Task 3020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3020 try 0 launched on executor multi with executor id 3021 parsl.dataflow.dflow INFO: Standard output for task 3020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/c61a67ac-3b9d-4cad-a97c-9030b77d142e_isrForCrosstalkSources_1125888_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/c61a67ac-3b9d-4cad-a97c-9030b77d142e_isrForCrosstalkSources_1125888_28.stderr parsl.dataflow.dflow DEBUG: Task 3021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3021 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3021 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3021 has memoization hash c475d21280f4e56bdaa72b856fb567c6 parsl.dataflow.memoization INFO: Task 3021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3021 try 0 launched on executor multi with executor id 3022 parsl.dataflow.dflow INFO: Standard output for task 3021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/4dbaa53b-2194-4a4b-aedf-dd0e213a875c_isrForCrosstalkSources_1125905_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/4dbaa53b-2194-4a4b-aedf-dd0e213a875c_isrForCrosstalkSources_1125905_16.stderr parsl.dataflow.dflow DEBUG: Task 3022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3022 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3022 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3022 has memoization hash a91c814d16b3eab440824e9280deb315 parsl.dataflow.memoization INFO: Task 3022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3022 try 0 launched on executor multi with executor id 3023 parsl.dataflow.dflow INFO: Standard output for task 3022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d37d63ec-0af3-4bf7-a2f9-b33fc66519dd_isrForCrosstalkSources_1125915_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d37d63ec-0af3-4bf7-a2f9-b33fc66519dd_isrForCrosstalkSources_1125915_58.stderr parsl.dataflow.dflow DEBUG: Task 3023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3023 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3023 has memoization hash 929a0c4903cca8d1ba883067aae2a11b parsl.dataflow.memoization INFO: Task 3023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393620c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3023 try 0 launched on executor multi with executor id 3024 parsl.dataflow.dflow INFO: Standard output for task 3023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/005411cc-84f0-48ea-9008-e428c9fb5278_isrForCrosstalkSources_1125891_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/005411cc-84f0-48ea-9008-e428c9fb5278_isrForCrosstalkSources_1125891_2.stderr parsl.dataflow.dflow DEBUG: Task 3024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3024 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3024 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3024 has memoization hash ccafc03cfcf0cf3e61a55c8372d2431d parsl.dataflow.memoization INFO: Task 3024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393631a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3024 try 0 launched on executor multi with executor id 3025 parsl.dataflow.dflow INFO: Standard output for task 3024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/3f696ac6-3c0c-40f4-bb00-9fbf8058121e_isrForCrosstalkSources_1125934_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/3f696ac6-3c0c-40f4-bb00-9fbf8058121e_isrForCrosstalkSources_1125934_26.stderr parsl.dataflow.dflow DEBUG: Task 3025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3025 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3025 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3025 has memoization hash 0b86e6a980d6439483d0aecd910b6223 parsl.dataflow.memoization INFO: Task 3025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3025 try 0 launched on executor multi with executor id 3026 parsl.dataflow.dflow INFO: Standard output for task 3025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/317839e1-6ee9-4c96-8a5c-770b9f5ed515_isrForCrosstalkSources_1125904_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/317839e1-6ee9-4c96-8a5c-770b9f5ed515_isrForCrosstalkSources_1125904_27.stderr parsl.dataflow.dflow DEBUG: Task 3026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3026 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3026 has memoization hash 46ea05d85593e44a4144c4ae42823cea parsl.dataflow.memoization INFO: Task 3026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3026 try 0 launched on executor multi with executor id 3027 parsl.dataflow.dflow INFO: Standard output for task 3026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/86d28b6a-71a6-4750-bd7f-463075c47f2e_isrForCrosstalkSources_1125928_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/86d28b6a-71a6-4750-bd7f-463075c47f2e_isrForCrosstalkSources_1125928_8.stderr parsl.dataflow.dflow DEBUG: Task 3027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3027 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3027 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3027 has memoization hash 8ddbb7a773feac5b41ced2c22a917a0f parsl.dataflow.memoization INFO: Task 3027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3027 try 0 launched on executor multi with executor id 3028 parsl.dataflow.dflow INFO: Standard output for task 3027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/02308bdb-75a7-434c-9f6c-6bfa2b521f70_isrForCrosstalkSources_1125918_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/02308bdb-75a7-434c-9f6c-6bfa2b521f70_isrForCrosstalkSources_1125918_52.stderr parsl.dataflow.dflow DEBUG: Task 3028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3028 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3028 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3028 has memoization hash 41819b0e1f55d75290692e8386ba3d1b parsl.dataflow.memoization INFO: Task 3028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3028 try 0 launched on executor multi with executor id 3029 parsl.dataflow.dflow INFO: Standard output for task 3028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/ebcb53d4-6052-4411-b9f9-e4c08bba7911_isrForCrosstalkSources_1125943_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/ebcb53d4-6052-4411-b9f9-e4c08bba7911_isrForCrosstalkSources_1125943_5.stderr parsl.dataflow.dflow DEBUG: Task 3029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3029 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3029 has memoization hash 3a0e79966ede4b6839539c165e6e1e87 parsl.dataflow.memoization INFO: Task 3029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3029 try 0 launched on executor multi with executor id 3030 parsl.dataflow.dflow INFO: Standard output for task 3029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/24ad4f41-104a-4c26-93e5-f4e1d4e618a2_isrForCrosstalkSources_1125915_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/24ad4f41-104a-4c26-93e5-f4e1d4e618a2_isrForCrosstalkSources_1125915_12.stderr parsl.dataflow.dflow DEBUG: Task 3030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3030 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3030 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3030 has memoization hash ecb2283c108ea4e45c0f0d16e6cab66d parsl.dataflow.memoization INFO: Task 3030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3030 try 0 launched on executor multi with executor id 3031 parsl.dataflow.dflow INFO: Standard output for task 3030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d1e787cd-9f68-4707-8784-2f0c9f46c170_isrForCrosstalkSources_1125911_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d1e787cd-9f68-4707-8784-2f0c9f46c170_isrForCrosstalkSources_1125911_43.stderr parsl.dataflow.dflow DEBUG: Task 3031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3031 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3031 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3031 has memoization hash 2e3032a57322d75902e2deed536de977 parsl.dataflow.memoization INFO: Task 3031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3031 try 0 launched on executor multi with executor id 3032 parsl.dataflow.dflow INFO: Standard output for task 3031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/da2b9ef5-6bdc-4ee1-be8c-3dd98e78629e_isrForCrosstalkSources_1125893_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/da2b9ef5-6bdc-4ee1-be8c-3dd98e78629e_isrForCrosstalkSources_1125893_48.stderr parsl.dataflow.dflow DEBUG: Task 3032 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3032 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3032 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3032 has memoization hash 2075d69f0036a9061a7cc207e900cb13 parsl.dataflow.memoization INFO: Task 3032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3032 try 0 launched on executor multi with executor id 3033 parsl.dataflow.dflow INFO: Standard output for task 3032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/dabdb0fb-6a27-4e2a-b79c-4aa2e0bac857_isrForCrosstalkSources_1125909_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/dabdb0fb-6a27-4e2a-b79c-4aa2e0bac857_isrForCrosstalkSources_1125909_14.stderr parsl.dataflow.dflow DEBUG: Task 3033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3033 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3033 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3033 has memoization hash d9eb054b9dd2ad5c63acaa9b2848e9c6 parsl.dataflow.memoization INFO: Task 3033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3033 try 0 launched on executor multi with executor id 3034 parsl.dataflow.dflow INFO: Standard output for task 3033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/950325fa-80f8-4e4c-9e6d-a581771e2758_isrForCrosstalkSources_1125925_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/950325fa-80f8-4e4c-9e6d-a581771e2758_isrForCrosstalkSources_1125925_50.stderr parsl.dataflow.dflow DEBUG: Task 3034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3034 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3034 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3034 has memoization hash a9b62d66c70ad5f6bb608454de5b406f parsl.dataflow.memoization INFO: Task 3034 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3034 try 0 launched on executor multi with executor id 3035 parsl.dataflow.dflow INFO: Standard output for task 3034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/98f88202-b26e-4c8a-b55a-162375bb156a_isrForCrosstalkSources_1125895_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/98f88202-b26e-4c8a-b55a-162375bb156a_isrForCrosstalkSources_1125895_50.stderr parsl.dataflow.dflow DEBUG: Task 3035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3035 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3035 has memoization hash c96bc06ad7ac4c26abceeb59097eb2b2 parsl.dataflow.memoization INFO: Task 3035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3035 try 0 launched on executor multi with executor id 3036 parsl.dataflow.dflow INFO: Standard output for task 3035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/9b64d7ce-3bd0-467f-8e45-c271f9af25c1_isrForCrosstalkSources_1125926_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/9b64d7ce-3bd0-467f-8e45-c271f9af25c1_isrForCrosstalkSources_1125926_36.stderr parsl.dataflow.dflow DEBUG: Task 3036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3036 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3036 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3036 has memoization hash 749f850092998f67a1aa8ab38a7e2b33 parsl.dataflow.memoization INFO: Task 3036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3036 try 0 launched on executor multi with executor id 3037 parsl.dataflow.dflow INFO: Standard output for task 3036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/e0c98b9e-9510-4d64-ba18-bc88cdc3ee89_isrForCrosstalkSources_1125897_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/e0c98b9e-9510-4d64-ba18-bc88cdc3ee89_isrForCrosstalkSources_1125897_28.stderr parsl.dataflow.dflow DEBUG: Task 3037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3037 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3037 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3037 has memoization hash 1167f2233555b16bb05282992a642d05 parsl.dataflow.memoization INFO: Task 3037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393628e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3037 try 0 launched on executor multi with executor id 3038 parsl.dataflow.dflow INFO: Standard output for task 3037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/1f20602e-98cc-4a8f-ae35-05e1ee7a6564_isrForCrosstalkSources_1125916_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/1f20602e-98cc-4a8f-ae35-05e1ee7a6564_isrForCrosstalkSources_1125916_32.stderr parsl.dataflow.dflow DEBUG: Task 3038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3038 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3038 has memoization hash 313caea5ef87808f1242c70be17c70d2 parsl.dataflow.memoization INFO: Task 3038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3038 try 0 launched on executor multi with executor id 3039 parsl.dataflow.dflow INFO: Standard output for task 3038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/aa987415-458d-48f0-a822-34e4d6a5923a_isrForCrosstalkSources_1125898_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/aa987415-458d-48f0-a822-34e4d6a5923a_isrForCrosstalkSources_1125898_16.stderr parsl.dataflow.dflow DEBUG: Task 3039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3039 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3039 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3039 has memoization hash 60d7f27acd334d49150f5583f805c58a parsl.dataflow.memoization INFO: Task 3039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393627a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3039 try 0 launched on executor multi with executor id 3040 parsl.dataflow.dflow INFO: Standard output for task 3039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/95921566-d44f-4edb-8844-c0c3fa4222ea_isrForCrosstalkSources_1125921_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/95921566-d44f-4edb-8844-c0c3fa4222ea_isrForCrosstalkSources_1125921_19.stderr parsl.dataflow.dflow DEBUG: Task 3040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3040 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3040 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3040 has memoization hash 673a7f0a13cc4b1714b9f853e37ce4a9 parsl.dataflow.memoization INFO: Task 3040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3040 try 0 launched on executor multi with executor id 3041 parsl.dataflow.dflow INFO: Standard output for task 3040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/7201b000-c288-4a89-964c-e52872890580_isrForCrosstalkSources_1125909_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/7201b000-c288-4a89-964c-e52872890580_isrForCrosstalkSources_1125909_61.stderr parsl.dataflow.dflow DEBUG: Task 3041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3041 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3041 has memoization hash d145e0ce7b83e0bda0cebc3de934233f parsl.dataflow.memoization INFO: Task 3041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3041 try 0 launched on executor multi with executor id 3042 parsl.dataflow.dflow INFO: Standard output for task 3041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/1f2d1dd2-1c47-4618-8772-dc94c99c6927_isrForCrosstalkSources_1125925_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/1f2d1dd2-1c47-4618-8772-dc94c99c6927_isrForCrosstalkSources_1125925_57.stderr parsl.dataflow.dflow DEBUG: Task 3042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3042 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3042 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3042 has memoization hash 953622437cb6bf0f8e4b20d77a2edad7 parsl.dataflow.memoization INFO: Task 3042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393625c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3042 try 0 launched on executor multi with executor id 3043 parsl.dataflow.dflow INFO: Standard output for task 3042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/bb1be7e1-1fd3-4f11-9a06-2a0b19a08138_isrForCrosstalkSources_1125890_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/bb1be7e1-1fd3-4f11-9a06-2a0b19a08138_isrForCrosstalkSources_1125890_50.stderr parsl.dataflow.dflow DEBUG: Task 3043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3043 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3043 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3043 has memoization hash 23a3e618de7fb0c92c7fd150933ca8a1 parsl.dataflow.memoization INFO: Task 3043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3043 try 0 launched on executor multi with executor id 3044 parsl.dataflow.dflow INFO: Standard output for task 3043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9d450f53-dc93-4f8f-8898-0b02aa291d55_isrForCrosstalkSources_1125921_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9d450f53-dc93-4f8f-8898-0b02aa291d55_isrForCrosstalkSources_1125921_53.stderr parsl.dataflow.dflow DEBUG: Task 3044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3044 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3044 has memoization hash 710a15d4596fd4f3304489558ee74bca parsl.dataflow.memoization INFO: Task 3044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3044 try 0 launched on executor multi with executor id 3045 parsl.dataflow.dflow INFO: Standard output for task 3044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/bb0230a7-fdf2-4371-87c2-4fe24caf7e44_isrForCrosstalkSources_1125928_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/bb0230a7-fdf2-4371-87c2-4fe24caf7e44_isrForCrosstalkSources_1125928_44.stderr parsl.dataflow.dflow DEBUG: Task 3045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3045 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3045 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3045 has memoization hash f4facb071d425b839d5d595b2650122b parsl.dataflow.memoization INFO: Task 3045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3045 try 0 launched on executor multi with executor id 3046 parsl.dataflow.dflow INFO: Standard output for task 3045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/aec9a431-5b71-4bd1-bc32-0e085561bb45_isrForCrosstalkSources_1125930_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/aec9a431-5b71-4bd1-bc32-0e085561bb45_isrForCrosstalkSources_1125930_19.stderr parsl.dataflow.dflow DEBUG: Task 3046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3046 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3046 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3046 has memoization hash e8b91f72003605e514f09efabd6e53d9 parsl.dataflow.memoization INFO: Task 3046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3046 try 0 launched on executor multi with executor id 3047 parsl.dataflow.dflow INFO: Standard output for task 3046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/cbb6e0a6-e401-4982-86c1-bc6c75540db9_isrForCrosstalkSources_1125899_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/cbb6e0a6-e401-4982-86c1-bc6c75540db9_isrForCrosstalkSources_1125899_20.stderr parsl.dataflow.dflow DEBUG: Task 3047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3047 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3047 has memoization hash 1b9a3db3c2c96bb720fa435ad1282e8a parsl.dataflow.memoization INFO: Task 3047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3047 try 0 launched on executor multi with executor id 3048 parsl.dataflow.dflow INFO: Standard output for task 3047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/f39ac24b-8358-4cbf-8202-8f6584bdd3ca_isrForCrosstalkSources_1125898_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/f39ac24b-8358-4cbf-8202-8f6584bdd3ca_isrForCrosstalkSources_1125898_40.stderr parsl.dataflow.dflow DEBUG: Task 3048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3048 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3048 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3048 has memoization hash c62ab84e09f96e938bb5c89e2900ab9f parsl.dataflow.memoization INFO: Task 3048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393623e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3048 try 0 launched on executor multi with executor id 3049 parsl.dataflow.dflow INFO: Standard output for task 3048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/23dda8d0-33db-450f-bda9-425e29967186_isrForCrosstalkSources_1125898_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/23dda8d0-33db-450f-bda9-425e29967186_isrForCrosstalkSources_1125898_33.stderr parsl.dataflow.dflow DEBUG: Task 3049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3049 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3049 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3049 has memoization hash 07807e96d5bd990adff867945f799e46 parsl.dataflow.memoization INFO: Task 3049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3049 try 0 launched on executor multi with executor id 3050 parsl.dataflow.dflow INFO: Standard output for task 3049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/82ebe9b7-c0de-45fc-a62c-340816580a08_isrForCrosstalkSources_1125890_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/82ebe9b7-c0de-45fc-a62c-340816580a08_isrForCrosstalkSources_1125890_48.stderr parsl.dataflow.dflow DEBUG: Task 3050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3050 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3050 has memoization hash a9ee0178873de630d289d4853c42e615 parsl.dataflow.memoization INFO: Task 3050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393622a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:04 proc_lsst.multi:170 [INFO] [multi] job_id 16896433 proc_lsst.multi INFO: [multi] job_id 16896433 2024-03-07 15:07:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 15:07:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:04 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896433 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896433 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 16896433 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-07 15:07:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-07 15:07:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 15:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3050 try 0 launched on executor multi with executor id 3051 parsl.dataflow.dflow INFO: Standard output for task 3050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/12300f5b-5c69-4055-9403-20e0758f375c_isrForCrosstalkSources_1125910_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/12300f5b-5c69-4055-9403-20e0758f375c_isrForCrosstalkSources_1125910_56.stderr parsl.dataflow.dflow DEBUG: Task 3051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3051 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3051 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3051 has memoization hash a6be2333d620683f1868d9add4b6008d parsl.dataflow.memoization INFO: Task 3051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3051 try 0 launched on executor multi with executor id 3052 parsl.dataflow.dflow INFO: Standard output for task 3051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/5f2947d9-2b44-4c26-bde9-a44490fbf8c7_isrForCrosstalkSources_1125917_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/5f2947d9-2b44-4c26-bde9-a44490fbf8c7_isrForCrosstalkSources_1125917_13.stderr parsl.dataflow.dflow DEBUG: Task 3052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3052 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3052 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3052 has memoization hash 5e0e3456ea9b279a41ed2b4d576466e9 parsl.dataflow.memoization INFO: Task 3052 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393636a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3052 try 0 launched on executor multi with executor id 3053 parsl.dataflow.dflow INFO: Standard output for task 3052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e911e6b5-5173-4fbe-9beb-e6c738287bed_isrForCrosstalkSources_1125929_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e911e6b5-5173-4fbe-9beb-e6c738287bed_isrForCrosstalkSources_1125929_40.stderr parsl.dataflow.dflow DEBUG: Task 3053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3053 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3053 has memoization hash 3a5d42d98ce93a32d238412a4a99b96d parsl.dataflow.memoization INFO: Task 3053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3053 try 0 launched on executor multi with executor id 3054 parsl.dataflow.dflow INFO: Standard output for task 3053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/2fa624e9-6cb4-4bfa-8b9d-09aceebd3a3f_isrForCrosstalkSources_1125913_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/2fa624e9-6cb4-4bfa-8b9d-09aceebd3a3f_isrForCrosstalkSources_1125913_38.stderr parsl.dataflow.dflow DEBUG: Task 3054 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3054 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3054 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3054 has memoization hash 6661e510ae6b916e88d369d9ed8f42cb parsl.dataflow.memoization INFO: Task 3054 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393616c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3054 try 0 launched on executor multi with executor id 3055 parsl.dataflow.dflow INFO: Standard output for task 3054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/85fce27e-56f4-4ad9-b449-a14dff4da330_isrForCrosstalkSources_1125913_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/85fce27e-56f4-4ad9-b449-a14dff4da330_isrForCrosstalkSources_1125913_35.stderr parsl.dataflow.dflow DEBUG: Task 3055 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3055 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3055 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3055 has memoization hash fa694e170c67bc11edb4e7fdc99d69cf parsl.dataflow.memoization INFO: Task 3055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393618a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3055 try 0 launched on executor multi with executor id 3056 parsl.dataflow.dflow INFO: Standard output for task 3055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/b8caaef0-fad4-4bc9-9ef1-b6146dcfbba6_isrForCrosstalkSources_1125912_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/b8caaef0-fad4-4bc9-9ef1-b6146dcfbba6_isrForCrosstalkSources_1125912_25.stderr parsl.dataflow.dflow DEBUG: Task 3056 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3056 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3056 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3056 has memoization hash ceb1684134a022be78e206ceee0c9b44 parsl.dataflow.memoization INFO: Task 3056 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a394c6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3056 try 0 launched on executor multi with executor id 3057 parsl.dataflow.dflow INFO: Standard output for task 3056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/c3717ad7-24f8-4f47-a364-0f67350a1fb4_isrForCrosstalkSources_1125896_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/c3717ad7-24f8-4f47-a364-0f67350a1fb4_isrForCrosstalkSources_1125896_27.stderr parsl.dataflow.dflow DEBUG: Task 3057 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3057 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3057 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3057 has memoization hash 4146a15314ec6be60686ca3ce967506f parsl.dataflow.memoization INFO: Task 3057 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393637e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3057 try 0 launched on executor multi with executor id 3058 parsl.dataflow.dflow INFO: Standard output for task 3057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c97825d7-9422-43d7-ad13-e36943e7bb36_isrForCrosstalkSources_1125893_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c97825d7-9422-43d7-ad13-e36943e7bb36_isrForCrosstalkSources_1125893_34.stderr parsl.dataflow.dflow DEBUG: Task 3058 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3058 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3058 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3058 has memoization hash b35204056f2ed76da5245eb4e6db12a2 parsl.dataflow.memoization INFO: Task 3058 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eaf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3058 try 0 launched on executor multi with executor id 3059 parsl.dataflow.dflow INFO: Standard output for task 3058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/57d0c219-0535-47ea-85b9-fe024c153abe_isrForCrosstalkSources_1125915_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/57d0c219-0535-47ea-85b9-fe024c153abe_isrForCrosstalkSources_1125915_33.stderr parsl.dataflow.dflow DEBUG: Task 3059 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3059 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3059 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3059 has memoization hash ca62f50bbb50b6878bc4103741fbc5ea parsl.dataflow.memoization INFO: Task 3059 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3059 try 0 launched on executor multi with executor id 3060 parsl.dataflow.dflow INFO: Standard output for task 3059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/4ae9b81d-9a26-4f8f-b93c-e8f62c2bfd9a_isrForCrosstalkSources_1125932_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/4ae9b81d-9a26-4f8f-b93c-e8f62c2bfd9a_isrForCrosstalkSources_1125932_31.stderr parsl.dataflow.dflow DEBUG: Task 3060 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3060 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3060 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3060 has memoization hash 3c349b21b251e202e66baefc0be6e682 parsl.dataflow.memoization INFO: Task 3060 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3060 try 0 launched on executor multi with executor id 3061 parsl.dataflow.dflow INFO: Standard output for task 3060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/03962a24-16dc-4f72-9172-b7aaef2cbe96_isrForCrosstalkSources_1125924_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/03962a24-16dc-4f72-9172-b7aaef2cbe96_isrForCrosstalkSources_1125924_61.stderr parsl.dataflow.dflow DEBUG: Task 3061 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3061 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3061 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3061 has memoization hash 219dc45f62ad8a6941e3708ad6d80775 parsl.dataflow.memoization INFO: Task 3061 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3061 try 0 launched on executor multi with executor id 3062 parsl.dataflow.dflow INFO: Standard output for task 3061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2bcb105a-e793-48c0-8cdc-ad40e73c141c_isrForCrosstalkSources_1125898_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2bcb105a-e793-48c0-8cdc-ad40e73c141c_isrForCrosstalkSources_1125898_27.stderr parsl.dataflow.dflow DEBUG: Task 3062 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3062 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3062 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3062 has memoization hash f566239fb2f1cae09ba5def063986e7d parsl.dataflow.memoization INFO: Task 3062 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3062 try 0 launched on executor multi with executor id 3063 parsl.dataflow.dflow INFO: Standard output for task 3062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/8460b97c-1a96-4512-879f-fc2629907178_isrForCrosstalkSources_1125924_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/8460b97c-1a96-4512-879f-fc2629907178_isrForCrosstalkSources_1125924_47.stderr parsl.dataflow.dflow DEBUG: Task 3063 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3063 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3063 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3063 has memoization hash 321e46d700286c5b555b97823f777d19 parsl.dataflow.memoization INFO: Task 3063 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393639c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3063 try 0 launched on executor multi with executor id 3064 parsl.dataflow.dflow INFO: Standard output for task 3063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/9bd7d441-38fd-413b-9a14-14260539302c_isrForCrosstalkSources_1125901_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/9bd7d441-38fd-413b-9a14-14260539302c_isrForCrosstalkSources_1125901_26.stderr parsl.dataflow.dflow DEBUG: Task 3064 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3064 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3064 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3064 has memoization hash e7f62c09b3f1954b2ba782d956d3e92c parsl.dataflow.memoization INFO: Task 3064 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3064 try 0 launched on executor multi with executor id 3065 parsl.dataflow.dflow INFO: Standard output for task 3064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/21bab288-0fef-4476-a515-f15a473fd675_isrForCrosstalkSources_1125917_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/21bab288-0fef-4476-a515-f15a473fd675_isrForCrosstalkSources_1125917_12.stderr parsl.dataflow.dflow DEBUG: Task 3065 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3065 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3065 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3065 has memoization hash 8000ffab4df8ae91242bcb0450ae2fe8 parsl.dataflow.memoization INFO: Task 3065 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3065 try 0 launched on executor multi with executor id 3066 parsl.dataflow.dflow INFO: Standard output for task 3065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/5b35b883-8d34-4a9b-af91-a1380fa7e53f_isrForCrosstalkSources_1125894_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/5b35b883-8d34-4a9b-af91-a1380fa7e53f_isrForCrosstalkSources_1125894_43.stderr parsl.dataflow.dflow DEBUG: Task 3066 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3066 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3066 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3066 has memoization hash 7c77ff81e9e9d4c5a1a187a32bea35a3 parsl.dataflow.memoization INFO: Task 3066 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ead40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3066 try 0 launched on executor multi with executor id 3067 parsl.dataflow.dflow INFO: Standard output for task 3066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/0aeea548-55d1-4580-be88-120930268e2b_isrForCrosstalkSources_1125901_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/0aeea548-55d1-4580-be88-120930268e2b_isrForCrosstalkSources_1125901_3.stderr parsl.dataflow.dflow DEBUG: Task 3067 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3067 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3067 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3067 has memoization hash 9936ccc4a836fa00250c141538fc9708 parsl.dataflow.memoization INFO: Task 3067 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3067 try 0 launched on executor multi with executor id 3068 parsl.dataflow.dflow INFO: Standard output for task 3067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/82006100-599d-436f-9240-0e7ae7a3be98_isrForCrosstalkSources_1125896_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/82006100-599d-436f-9240-0e7ae7a3be98_isrForCrosstalkSources_1125896_25.stderr parsl.dataflow.dflow DEBUG: Task 3068 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3068 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3068 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3068 has memoization hash 13e53889250fd0854a4f57b1467ed728 parsl.dataflow.memoization INFO: Task 3068 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3068 try 0 launched on executor multi with executor id 3069 parsl.dataflow.dflow INFO: Standard output for task 3068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/6c65b762-535d-4d28-baa2-c923ba5f90d2_isrForCrosstalkSources_1125912_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/6c65b762-535d-4d28-baa2-c923ba5f90d2_isrForCrosstalkSources_1125912_42.stderr parsl.dataflow.dflow DEBUG: Task 3069 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3069 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3069 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3069 has memoization hash 3d48ee7cab664638872c795df328741d parsl.dataflow.memoization INFO: Task 3069 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3069 try 0 launched on executor multi with executor id 3070 parsl.dataflow.dflow INFO: Standard output for task 3069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/40f665f9-5aed-4d14-b401-3c2aee56d183_isrForCrosstalkSources_1125918_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/40f665f9-5aed-4d14-b401-3c2aee56d183_isrForCrosstalkSources_1125918_13.stderr parsl.dataflow.dflow DEBUG: Task 3070 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3070 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3070 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3070 has memoization hash fe70fd2738aa559b7b82c75ea00bd300 parsl.dataflow.memoization INFO: Task 3070 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3070 try 0 launched on executor multi with executor id 3071 parsl.dataflow.dflow INFO: Standard output for task 3070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/0c7ce43e-9bef-43b7-aead-542bcdf9dc79_isrForCrosstalkSources_1125944_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/0c7ce43e-9bef-43b7-aead-542bcdf9dc79_isrForCrosstalkSources_1125944_25.stderr parsl.dataflow.dflow DEBUG: Task 3071 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3071 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3071 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3071 has memoization hash cf4451a1755d65db0ed94a471638da85 parsl.dataflow.memoization INFO: Task 3071 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3071 try 0 launched on executor multi with executor id 3072 parsl.dataflow.dflow INFO: Standard output for task 3071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/950b802a-3bad-40f8-b699-e5f3976a3254_isrForCrosstalkSources_1125944_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/950b802a-3bad-40f8-b699-e5f3976a3254_isrForCrosstalkSources_1125944_45.stderr parsl.dataflow.dflow DEBUG: Task 3072 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3072 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3072 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3072 has memoization hash 4c59256d1d35e8bff3b13efc49b0f94c parsl.dataflow.memoization INFO: Task 3072 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3072 try 0 launched on executor multi with executor id 3073 parsl.dataflow.dflow INFO: Standard output for task 3072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/44d03f63-af1b-4744-b2ac-ea146518a009_isrForCrosstalkSources_1125943_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/44d03f63-af1b-4744-b2ac-ea146518a009_isrForCrosstalkSources_1125943_54.stderr parsl.dataflow.dflow DEBUG: Task 3073 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3073 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3073 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3073 has memoization hash 13fd20988c43a6d92b30f370c18335dd parsl.dataflow.memoization INFO: Task 3073 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3073 try 0 launched on executor multi with executor id 3074 parsl.dataflow.dflow INFO: Standard output for task 3073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/1044760c-22c4-4c80-8b89-98f9b265963c_isrForCrosstalkSources_1125941_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/1044760c-22c4-4c80-8b89-98f9b265963c_isrForCrosstalkSources_1125941_45.stderr parsl.dataflow.dflow DEBUG: Task 3074 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3074 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3074 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3074 has memoization hash 1e9457905764f3ec22024eb5268a161c parsl.dataflow.memoization INFO: Task 3074 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3074 try 0 launched on executor multi with executor id 3075 parsl.dataflow.dflow INFO: Standard output for task 3074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/3bed326d-313e-4699-a82e-871ea5d84569_isrForCrosstalkSources_1125908_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/3bed326d-313e-4699-a82e-871ea5d84569_isrForCrosstalkSources_1125908_46.stderr parsl.dataflow.dflow DEBUG: Task 3075 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3075 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3075 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3075 has memoization hash e122e6863af684ec30bb9da78cd66922 parsl.dataflow.memoization INFO: Task 3075 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3075 try 0 launched on executor multi with executor id 3076 parsl.dataflow.dflow INFO: Standard output for task 3075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/2a015193-b520-4208-b592-f470c25e29f8_isrForCrosstalkSources_1125888_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/2a015193-b520-4208-b592-f470c25e29f8_isrForCrosstalkSources_1125888_38.stderr parsl.dataflow.dflow DEBUG: Task 3076 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3076 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3076 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3076 has memoization hash f0a7afc3839e66fd8b067b3aac3909a7 parsl.dataflow.memoization INFO: Task 3076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3076 try 0 launched on executor multi with executor id 3077 parsl.dataflow.dflow INFO: Standard output for task 3076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8642ae0a-760c-48ad-8a19-fd33115154c0_isrForCrosstalkSources_1125936_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8642ae0a-760c-48ad-8a19-fd33115154c0_isrForCrosstalkSources_1125936_2.stderr parsl.dataflow.dflow DEBUG: Task 3077 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3077 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3077 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3077 has memoization hash 9d5de52a6c288185d89b68ec37764eac parsl.dataflow.memoization INFO: Task 3077 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3077 try 0 launched on executor multi with executor id 3078 parsl.dataflow.dflow INFO: Standard output for task 3077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/49063da9-9778-46e4-b617-17f2c25c8316_isrForCrosstalkSources_1125919_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/49063da9-9778-46e4-b617-17f2c25c8316_isrForCrosstalkSources_1125919_42.stderr parsl.dataflow.dflow DEBUG: Task 3078 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3078 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3078 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3078 has memoization hash 38afaddc6ebe24211d82a8f15aee158f parsl.dataflow.memoization INFO: Task 3078 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3078 try 0 launched on executor multi with executor id 3079 parsl.dataflow.dflow INFO: Standard output for task 3078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/6dd377c8-d2dd-4b48-8d03-3a09e8167fee_isrForCrosstalkSources_1125936_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/6dd377c8-d2dd-4b48-8d03-3a09e8167fee_isrForCrosstalkSources_1125936_8.stderr parsl.dataflow.dflow DEBUG: Task 3079 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3079 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3079 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3079 has memoization hash 36c55e94f3fa4a2f8a019195890ecd15 parsl.dataflow.memoization INFO: Task 3079 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3079 try 0 launched on executor multi with executor id 3080 parsl.dataflow.dflow INFO: Standard output for task 3079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/dbb17ed1-e904-4cc3-8df7-b7b4885913fd_isrForCrosstalkSources_1125939_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/dbb17ed1-e904-4cc3-8df7-b7b4885913fd_isrForCrosstalkSources_1125939_10.stderr parsl.dataflow.dflow DEBUG: Task 3080 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3080 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3080 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3080 has memoization hash db47d44df2b065ecc24cbee095d85f4e parsl.dataflow.memoization INFO: Task 3080 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3080 try 0 launched on executor multi with executor id 3081 parsl.dataflow.dflow INFO: Standard output for task 3080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/cc32b263-8b16-4074-8205-74731d3bf67a_isrForCrosstalkSources_1125925_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/cc32b263-8b16-4074-8205-74731d3bf67a_isrForCrosstalkSources_1125925_37.stderr parsl.dataflow.dflow DEBUG: Task 3081 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3081 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3081 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3081 has memoization hash 54b587df18f691953c874bd8b15ce185 parsl.dataflow.memoization INFO: Task 3081 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3081 try 0 launched on executor multi with executor id 3082 parsl.dataflow.dflow INFO: Standard output for task 3081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/446332c1-991e-497d-9cb3-7fd78d0b2e57_isrForCrosstalkSources_1125940_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/446332c1-991e-497d-9cb3-7fd78d0b2e57_isrForCrosstalkSources_1125940_48.stderr parsl.dataflow.dflow DEBUG: Task 3082 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3082 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3082 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3082 has memoization hash db21a4063c6b26eb6c297ed6994c5d2c parsl.dataflow.memoization INFO: Task 3082 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3082 try 0 launched on executor multi with executor id 3083 parsl.dataflow.dflow INFO: Standard output for task 3082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e8d27326-be9e-4653-a7ac-cd3a34f9277e_isrForCrosstalkSources_1125901_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e8d27326-be9e-4653-a7ac-cd3a34f9277e_isrForCrosstalkSources_1125901_44.stderr parsl.dataflow.dflow DEBUG: Task 3083 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3083 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3083 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3083 has memoization hash 0d391a8deaca11070e8f308cb4185326 parsl.dataflow.memoization INFO: Task 3083 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3083 try 0 launched on executor multi with executor id 3084 parsl.dataflow.dflow INFO: Standard output for task 3083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/cd1ba6bf-b2e3-4e9a-981a-49a892acc54c_isrForCrosstalkSources_1125922_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/cd1ba6bf-b2e3-4e9a-981a-49a892acc54c_isrForCrosstalkSources_1125922_13.stderr parsl.dataflow.dflow DEBUG: Task 3084 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3084 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3084 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3084 has memoization hash 231b87498a9fe27a2e6ebd2afd6ab9aa parsl.dataflow.memoization INFO: Task 3084 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3084 try 0 launched on executor multi with executor id 3085 parsl.dataflow.dflow INFO: Standard output for task 3084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/b6dfd2bb-c62a-4bd0-9878-f12f92d2fba6_isrForCrosstalkSources_1125902_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/b6dfd2bb-c62a-4bd0-9878-f12f92d2fba6_isrForCrosstalkSources_1125902_51.stderr parsl.dataflow.dflow DEBUG: Task 3085 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3085 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3085 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3085 has memoization hash 498bc7587dbf23a4ee1fca53624e8e5e parsl.dataflow.memoization INFO: Task 3085 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3085 try 0 launched on executor multi with executor id 3086 parsl.dataflow.dflow INFO: Standard output for task 3085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/ddf8da26-68ea-4ebc-9993-c4419839443c_isrForCrosstalkSources_1125941_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/ddf8da26-68ea-4ebc-9993-c4419839443c_isrForCrosstalkSources_1125941_57.stderr 2024-03-07 15:07:04 proc_lsst.multi:170 [INFO] [multi] job_id 16896434 proc_lsst.multi INFO: [multi] job_id 16896434 2024-03-07 15:07:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 15:07:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:04 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896434 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896434 parsl.dataflow.dflow DEBUG: Task 3086 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3086 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3086 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3086 has memoization hash 4a40e31ba4597dea0fb9cda29d355afc parsl.dataflow.memoization INFO: Task 3086 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 16896434 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-07 15:07:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-07 15:07:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 15:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3086 try 0 launched on executor multi with executor id 3087 parsl.dataflow.dflow INFO: Standard output for task 3086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/dff9bda1-5fde-40b8-bbd7-26d5179fd45e_isrForCrosstalkSources_1125943_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/dff9bda1-5fde-40b8-bbd7-26d5179fd45e_isrForCrosstalkSources_1125943_17.stderr parsl.dataflow.dflow DEBUG: Task 3087 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3087 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3087 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3087 has memoization hash 067d3a429920c2f6f2677387bc054621 parsl.dataflow.memoization INFO: Task 3087 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3087 try 0 launched on executor multi with executor id 3088 parsl.dataflow.dflow INFO: Standard output for task 3087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/0866168f-f4b4-4a98-a93c-7f318145c598_isrForCrosstalkSources_1125936_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/0866168f-f4b4-4a98-a93c-7f318145c598_isrForCrosstalkSources_1125936_37.stderr parsl.dataflow.dflow DEBUG: Task 3088 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3088 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3088 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3088 has memoization hash 22c184e95270b09d0ebd0ec97be230ef parsl.dataflow.memoization INFO: Task 3088 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39361580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3088 try 0 launched on executor multi with executor id 3089 parsl.dataflow.dflow INFO: Standard output for task 3088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/4b976a96-1d24-452e-b40a-1c1b0d3b10db_isrForCrosstalkSources_1125902_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/4b976a96-1d24-452e-b40a-1c1b0d3b10db_isrForCrosstalkSources_1125902_28.stderr parsl.dataflow.dflow DEBUG: Task 3089 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3089 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3089 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3089 has memoization hash 82c8eb7bb0c72ab43ebf029b92ccf042 parsl.dataflow.memoization INFO: Task 3089 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39362020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3089 try 0 launched on executor multi with executor id 3090 parsl.dataflow.dflow INFO: Standard output for task 3089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/4da0ff6f-ab07-4b2e-8a31-1aea1738c0dc_isrForCrosstalkSources_1125902_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/4da0ff6f-ab07-4b2e-8a31-1aea1738c0dc_isrForCrosstalkSources_1125902_49.stderr parsl.dataflow.dflow DEBUG: Task 3090 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3090 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3090 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3090 has memoization hash 227508cd19086f6b0a713b59fc9dbec1 parsl.dataflow.memoization INFO: Task 3090 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3090 try 0 launched on executor multi with executor id 3091 parsl.dataflow.dflow INFO: Standard output for task 3090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/02120c32-ed15-4771-ab1a-e8ad2fc1ded7_isrForCrosstalkSources_1125938_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/02120c32-ed15-4771-ab1a-e8ad2fc1ded7_isrForCrosstalkSources_1125938_32.stderr parsl.dataflow.dflow DEBUG: Task 3091 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3091 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3091 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3091 has memoization hash 29264b75c5e696f1f2b34f0329f8a66d parsl.dataflow.memoization INFO: Task 3091 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3091 try 0 launched on executor multi with executor id 3092 parsl.dataflow.dflow INFO: Standard output for task 3091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/6ccb47d8-0ea5-4208-9984-85b9a358aa04_isrForCrosstalkSources_1125946_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/6ccb47d8-0ea5-4208-9984-85b9a358aa04_isrForCrosstalkSources_1125946_42.stderr parsl.dataflow.dflow DEBUG: Task 3092 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3092 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3092 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3092 has memoization hash f57e4521ecc7deb05214e1c53d97b506 parsl.dataflow.memoization INFO: Task 3092 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3092 try 0 launched on executor multi with executor id 3093 parsl.dataflow.dflow INFO: Standard output for task 3092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/893029c4-329c-47f7-bcf5-7bf37becda87_isrForCrosstalkSources_1125917_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/893029c4-329c-47f7-bcf5-7bf37becda87_isrForCrosstalkSources_1125917_59.stderr parsl.dataflow.dflow DEBUG: Task 3093 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3093 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3093 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3093 has memoization hash 28fc0ec692d6b45a93cad62ccf3a0291 parsl.dataflow.memoization INFO: Task 3093 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3093 try 0 launched on executor multi with executor id 3094 parsl.dataflow.dflow INFO: Standard output for task 3093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/5d9e82e7-5c11-49c6-936c-502ed1830c1c_isrForCrosstalkSources_1125901_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/5d9e82e7-5c11-49c6-936c-502ed1830c1c_isrForCrosstalkSources_1125901_5.stderr parsl.dataflow.dflow DEBUG: Task 3094 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3094 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3094 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3094 has memoization hash 7c743cad559fed6db3af13144dc559ce parsl.dataflow.memoization INFO: Task 3094 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3094 try 0 launched on executor multi with executor id 3095 parsl.dataflow.dflow INFO: Standard output for task 3094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/d1d21181-9921-4a51-8e33-d2cacd80a7a6_isrForCrosstalkSources_1125924_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/d1d21181-9921-4a51-8e33-d2cacd80a7a6_isrForCrosstalkSources_1125924_31.stderr parsl.dataflow.dflow DEBUG: Task 3095 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3095 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3095 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3095 has memoization hash 2b828ed01fe61b8223efdab8aa61e740 parsl.dataflow.memoization INFO: Task 3095 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ebe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3095 try 0 launched on executor multi with executor id 3096 parsl.dataflow.dflow INFO: Standard output for task 3095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/7ea372c0-506f-4b56-8d09-4276fcae9f94_isrForCrosstalkSources_1125906_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/7ea372c0-506f-4b56-8d09-4276fcae9f94_isrForCrosstalkSources_1125906_43.stderr parsl.dataflow.dflow DEBUG: Task 3096 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3096 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3096 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3096 has memoization hash c898a273fcb1896f0257d2e060f41941 parsl.dataflow.memoization INFO: Task 3096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3096 try 0 launched on executor multi with executor id 3097 parsl.dataflow.dflow INFO: Standard output for task 3096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/48a779a5-10c5-4e56-ba7c-e18e784de7a6_isrForCrosstalkSources_1125931_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/48a779a5-10c5-4e56-ba7c-e18e784de7a6_isrForCrosstalkSources_1125931_12.stderr parsl.dataflow.dflow DEBUG: Task 3097 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3097 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3097 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3097 has memoization hash d256e9e919d6ad16429ed4b6c53add03 parsl.dataflow.memoization INFO: Task 3097 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ebd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3097 try 0 launched on executor multi with executor id 3098 parsl.dataflow.dflow INFO: Standard output for task 3097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/4f857cf6-50cc-4b0d-a02c-0a38473481ee_isrForCrosstalkSources_1125947_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/4f857cf6-50cc-4b0d-a02c-0a38473481ee_isrForCrosstalkSources_1125947_44.stderr parsl.dataflow.dflow DEBUG: Task 3098 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3098 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3098 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3098 has memoization hash 12e93a0ce7fec327c6450b50f1f2e14f parsl.dataflow.memoization INFO: Task 3098 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ebec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3098 try 0 launched on executor multi with executor id 3099 parsl.dataflow.dflow INFO: Standard output for task 3098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/a961ecf8-8c8e-4d81-9c72-8ab5f196ce36_isrForCrosstalkSources_1125897_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/a961ecf8-8c8e-4d81-9c72-8ab5f196ce36_isrForCrosstalkSources_1125897_49.stderr parsl.dataflow.dflow DEBUG: Task 3099 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3099 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3099 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3099 has memoization hash 46924675d2ffe5355a136010fd23e24f parsl.dataflow.memoization INFO: Task 3099 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ebce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3099 try 0 launched on executor multi with executor id 3100 parsl.dataflow.dflow INFO: Standard output for task 3099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/fb9d3df2-3b65-497c-995c-e2c92658ce4f_isrForCrosstalkSources_1125888_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/fb9d3df2-3b65-497c-995c-e2c92658ce4f_isrForCrosstalkSources_1125888_41.stderr parsl.dataflow.dflow DEBUG: Task 3100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3100 has memoization hash 0126d25ab7464d4684f4d6f120ebd9c9 parsl.dataflow.memoization INFO: Task 3100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ebc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3100 try 0 launched on executor multi with executor id 3101 parsl.dataflow.dflow INFO: Standard output for task 3100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/527fa5e0-3b80-4a2f-a16a-4f752d1628e3_isrForCrosstalkSources_1125903_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/527fa5e0-3b80-4a2f-a16a-4f752d1628e3_isrForCrosstalkSources_1125903_56.stderr parsl.dataflow.dflow DEBUG: Task 3101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3101 has memoization hash 740c453a9441303468b0e18f73a1568b parsl.dataflow.memoization INFO: Task 3101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ebba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3101 try 0 launched on executor multi with executor id 3102 parsl.dataflow.dflow INFO: Standard output for task 3101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/1dfa2744-1cfd-4fd2-a666-86ed0ca7dbdf_isrForCrosstalkSources_1125918_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/1dfa2744-1cfd-4fd2-a666-86ed0ca7dbdf_isrForCrosstalkSources_1125918_44.stderr parsl.dataflow.dflow DEBUG: Task 3102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3102 has memoization hash 92f1e6298a914530f5951ce4686bc817 parsl.dataflow.memoization INFO: Task 3102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3102 try 0 launched on executor multi with executor id 3103 parsl.dataflow.dflow INFO: Standard output for task 3102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/be987c70-4555-4749-8c1d-dcff94f20002_isrForCrosstalkSources_1125905_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/be987c70-4555-4749-8c1d-dcff94f20002_isrForCrosstalkSources_1125905_49.stderr parsl.dataflow.dflow DEBUG: Task 3103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3103 has memoization hash 58931a4268e803b3cb5ada545686fd37 parsl.dataflow.memoization INFO: Task 3103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3103 try 0 launched on executor multi with executor id 3104 parsl.dataflow.dflow INFO: Standard output for task 3103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/49a43ef3-48c1-4080-8e7a-f9fbb2efb66e_isrForCrosstalkSources_1125896_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/49a43ef3-48c1-4080-8e7a-f9fbb2efb66e_isrForCrosstalkSources_1125896_5.stderr parsl.dataflow.dflow DEBUG: Task 3104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3104 has memoization hash 044dca2656fd36543bc4d51e3e91e8a8 parsl.dataflow.memoization INFO: Task 3104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3104 try 0 launched on executor multi with executor id 3105 parsl.dataflow.dflow INFO: Standard output for task 3104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/2704ca30-2737-49c8-ba33-2747f5a7cc35_isrForCrosstalkSources_1125892_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/2704ca30-2737-49c8-ba33-2747f5a7cc35_isrForCrosstalkSources_1125892_20.stderr parsl.dataflow.dflow DEBUG: Task 3105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3105 has memoization hash 28c4c479c011dd4dcb6aeb16cb159835 parsl.dataflow.memoization INFO: Task 3105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3105 try 0 launched on executor multi with executor id 3106 parsl.dataflow.dflow INFO: Standard output for task 3105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/80296b39-c113-4fbc-a833-4f9773557c80_isrForCrosstalkSources_1125914_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/80296b39-c113-4fbc-a833-4f9773557c80_isrForCrosstalkSources_1125914_44.stderr parsl.dataflow.dflow DEBUG: Task 3106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3106 has memoization hash a9b7987d6c65b8db5b301df7cfa2c073 parsl.dataflow.memoization INFO: Task 3106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3106 try 0 launched on executor multi with executor id 3107 parsl.dataflow.dflow INFO: Standard output for task 3106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/f26d2f3d-5610-472a-b05e-b458ed4740f2_isrForCrosstalkSources_1125947_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/f26d2f3d-5610-472a-b05e-b458ed4740f2_isrForCrosstalkSources_1125947_19.stderr parsl.dataflow.dflow DEBUG: Task 3107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3107 has memoization hash 3afc3af3055d4a663c162f8de965b2f8 parsl.dataflow.memoization INFO: Task 3107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3107 try 0 launched on executor multi with executor id 3108 parsl.dataflow.dflow INFO: Standard output for task 3107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/b365f0f0-8555-4d7d-8244-9fdbfae08e72_isrForCrosstalkSources_1125902_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/b365f0f0-8555-4d7d-8244-9fdbfae08e72_isrForCrosstalkSources_1125902_26.stderr parsl.dataflow.dflow DEBUG: Task 3108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3108 has memoization hash 622abd670b06e340b8b5c3281283e6ef parsl.dataflow.memoization INFO: Task 3108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3108 try 0 launched on executor multi with executor id 3109 parsl.dataflow.dflow INFO: Standard output for task 3108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/ce20d434-50e7-4c59-be78-41c0cc4b578c_isrForCrosstalkSources_1125932_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/ce20d434-50e7-4c59-be78-41c0cc4b578c_isrForCrosstalkSources_1125932_2.stderr parsl.dataflow.dflow DEBUG: Task 3109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3109 has memoization hash 5ae026d7d0f523bf7af19b8385fc7b32 parsl.dataflow.memoization INFO: Task 3109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3109 try 0 launched on executor multi with executor id 3110 parsl.dataflow.dflow INFO: Standard output for task 3109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/c4f0071d-bf1f-4d5a-aee6-b7b2d530cf88_isrForCrosstalkSources_1125942_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/c4f0071d-bf1f-4d5a-aee6-b7b2d530cf88_isrForCrosstalkSources_1125942_43.stderr parsl.dataflow.dflow DEBUG: Task 3110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3110 has memoization hash 61be17bc2dd3bf26a6e32f14c6279fbd parsl.dataflow.memoization INFO: Task 3110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3110 try 0 launched on executor multi with executor id 3111 parsl.dataflow.dflow INFO: Standard output for task 3110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/38be1731-f94d-4cf0-9aae-cf5599c232fc_isrForCrosstalkSources_1125909_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/38be1731-f94d-4cf0-9aae-cf5599c232fc_isrForCrosstalkSources_1125909_44.stderr parsl.dataflow.dflow DEBUG: Task 3111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3111 has memoization hash 9902de824462247298a455e4a44fbd73 parsl.dataflow.memoization INFO: Task 3111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3111 try 0 launched on executor multi with executor id 3112 parsl.dataflow.dflow INFO: Standard output for task 3111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/895be516-ed03-4331-a6c9-70ee0b8de0e1_isrForCrosstalkSources_1125941_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/895be516-ed03-4331-a6c9-70ee0b8de0e1_isrForCrosstalkSources_1125941_60.stderr parsl.dataflow.dflow DEBUG: Task 3112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3112 has memoization hash ae065cb7361d9cbde2032126c2bd1e39 parsl.dataflow.memoization INFO: Task 3112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3112 try 0 launched on executor multi with executor id 3113 parsl.dataflow.dflow INFO: Standard output for task 3112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/94194ae5-291b-46c3-a0cc-140438cda44f_isrForCrosstalkSources_1125909_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/94194ae5-291b-46c3-a0cc-140438cda44f_isrForCrosstalkSources_1125909_54.stderr parsl.dataflow.dflow DEBUG: Task 3113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3113 has memoization hash 45441fa33ab0d755c4d1c32ed49abf43 parsl.dataflow.memoization INFO: Task 3113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3113 try 0 launched on executor multi with executor id 3114 parsl.dataflow.dflow INFO: Standard output for task 3113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/7385d986-7223-495a-8474-577680892f9c_isrForCrosstalkSources_1125923_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/7385d986-7223-495a-8474-577680892f9c_isrForCrosstalkSources_1125923_42.stderr parsl.dataflow.dflow DEBUG: Task 3114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3114 has memoization hash 8564ccd1fde57e6d9ba60fe813271f74 parsl.dataflow.memoization INFO: Task 3114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3114 try 0 launched on executor multi with executor id 3115 parsl.dataflow.dflow INFO: Standard output for task 3114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/db83d08f-324e-43b8-801d-758eef971c99_isrForCrosstalkSources_1125922_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/db83d08f-324e-43b8-801d-758eef971c99_isrForCrosstalkSources_1125922_2.stderr parsl.dataflow.dflow DEBUG: Task 3115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3115 has memoization hash 26375d38ca2780a860996b38144258d4 parsl.dataflow.memoization INFO: Task 3115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3115 try 0 launched on executor multi with executor id 3116 parsl.dataflow.dflow INFO: Standard output for task 3115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/b5ef6a17-7fb6-4db5-a154-0d3030956cb6_isrForCrosstalkSources_1125897_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/b5ef6a17-7fb6-4db5-a154-0d3030956cb6_isrForCrosstalkSources_1125897_43.stderr parsl.dataflow.dflow DEBUG: Task 3116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3116 has memoization hash f7d012bce2b275d66610685c60d8355f parsl.dataflow.memoization INFO: Task 3116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3116 try 0 launched on executor multi with executor id 3117 parsl.dataflow.dflow INFO: Standard output for task 3116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/d1812e71-5bf5-4dc7-ae35-6ceaf6ca6c26_isrForCrosstalkSources_1125895_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/d1812e71-5bf5-4dc7-ae35-6ceaf6ca6c26_isrForCrosstalkSources_1125895_37.stderr parsl.dataflow.dflow DEBUG: Task 3117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3117 has memoization hash 29701ae6ef1e8bebe0ac89f563819b4c parsl.dataflow.memoization INFO: Task 3117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3117 try 0 launched on executor multi with executor id 3118 parsl.dataflow.dflow INFO: Standard output for task 3117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/0866482d-5c6d-4e2f-8bac-1c97f187e91a_isrForCrosstalkSources_1125924_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/0866482d-5c6d-4e2f-8bac-1c97f187e91a_isrForCrosstalkSources_1125924_9.stderr parsl.dataflow.dflow DEBUG: Task 3118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3118 has memoization hash 937960950fb8a094e738dc7568f7a343 parsl.dataflow.memoization INFO: Task 3118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3118 try 0 launched on executor multi with executor id 3119 parsl.dataflow.dflow INFO: Standard output for task 3118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/5ae12768-381b-40f4-ac84-1b8adc9ed2df_isrForCrosstalkSources_1125897_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/5ae12768-381b-40f4-ac84-1b8adc9ed2df_isrForCrosstalkSources_1125897_12.stderr parsl.dataflow.dflow DEBUG: Task 3119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3119 has memoization hash f7db97415d69baced8a037b4403d3ffd parsl.dataflow.memoization INFO: Task 3119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3119 try 0 launched on executor multi with executor id 3120 parsl.dataflow.dflow INFO: Standard output for task 3119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/632e83ed-c15c-42c6-8e53-30aef12f2451_isrForCrosstalkSources_1125911_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/632e83ed-c15c-42c6-8e53-30aef12f2451_isrForCrosstalkSources_1125911_22.stderr parsl.dataflow.dflow DEBUG: Task 3120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3120 has memoization hash ad3578d71d359b08506b50c82cad9a85 parsl.dataflow.memoization INFO: Task 3120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3120 try 0 launched on executor multi with executor id 3121 parsl.dataflow.dflow INFO: Standard output for task 3120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/9bc06530-61e2-4d32-ba7c-e33cf41270e6_isrForCrosstalkSources_1125900_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/9bc06530-61e2-4d32-ba7c-e33cf41270e6_isrForCrosstalkSources_1125900_58.stderr parsl.dataflow.dflow DEBUG: Task 3121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3121 has memoization hash b25a638d59e7e5b24944aeee086e4b6f parsl.dataflow.memoization INFO: Task 3121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3121 try 0 launched on executor multi with executor id 3122 parsl.dataflow.dflow INFO: Standard output for task 3121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/a1831642-bf3a-4716-a166-9a35ecb841c6_isrForCrosstalkSources_1125944_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/a1831642-bf3a-4716-a166-9a35ecb841c6_isrForCrosstalkSources_1125944_42.stderr parsl.dataflow.dflow DEBUG: Task 3122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3122 has memoization hash 95519ab8db2403d1c822736742fd9164 parsl.dataflow.memoization INFO: Task 3122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ea980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3122 try 0 launched on executor multi with executor id 3123 parsl.dataflow.dflow INFO: Standard output for task 3122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/f9f70594-2e10-40db-b8c1-594d890f5388_isrForCrosstalkSources_1125937_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/f9f70594-2e10-40db-b8c1-594d890f5388_isrForCrosstalkSources_1125937_14.stderr parsl.dataflow.dflow DEBUG: Task 3123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3123 has memoization hash fc81f36def681a4cb54852a23bc1e397 parsl.dataflow.memoization INFO: Task 3123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3123 try 0 launched on executor multi with executor id 3124 parsl.dataflow.dflow INFO: Standard output for task 3123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/c0339678-fa18-47cb-a785-8c0d838b5c19_isrForCrosstalkSources_1125935_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/c0339678-fa18-47cb-a785-8c0d838b5c19_isrForCrosstalkSources_1125935_6.stderr parsl.dataflow.dflow DEBUG: Task 3124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3124 has memoization hash 2bcf611d2bee15f9dad62ac7adee7675 parsl.dataflow.memoization INFO: Task 3124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ea8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3124 try 0 launched on executor multi with executor id 3125 parsl.dataflow.dflow INFO: Standard output for task 3124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/73a97a49-ade6-4bbd-abb3-89415db73ca3_isrForCrosstalkSources_1125939_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/73a97a49-ade6-4bbd-abb3-89415db73ca3_isrForCrosstalkSources_1125939_33.stderr parsl.dataflow.dflow DEBUG: Task 3125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3125 has memoization hash 42f0721c9b35634c08ee477f258d0d50 parsl.dataflow.memoization INFO: Task 3125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ea840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3125 try 0 launched on executor multi with executor id 3126 parsl.dataflow.dflow INFO: Standard output for task 3125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/c48efe97-7fb0-48ba-8e28-da84027fb4ab_isrForCrosstalkSources_1125939_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/c48efe97-7fb0-48ba-8e28-da84027fb4ab_isrForCrosstalkSources_1125939_62.stderr parsl.dataflow.dflow DEBUG: Task 3126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3126 has memoization hash 393ac875c1ce491f4b31ee4263648f5d parsl.dataflow.memoization INFO: Task 3126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3126 try 0 launched on executor multi with executor id 3127 parsl.dataflow.dflow INFO: Standard output for task 3126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/0b8b7046-476e-486f-88c1-d095642c5734_isrForCrosstalkSources_1125915_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/0b8b7046-476e-486f-88c1-d095642c5734_isrForCrosstalkSources_1125915_56.stderr parsl.dataflow.dflow DEBUG: Task 3127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3127 has memoization hash 4364d602be38f92617a20f952becd275 parsl.dataflow.memoization INFO: Task 3127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ea7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3127 try 0 launched on executor multi with executor id 3128 parsl.dataflow.dflow INFO: Standard output for task 3127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/0ab5f648-0de8-46be-9076-052c68e804bd_isrForCrosstalkSources_1125900_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/0ab5f648-0de8-46be-9076-052c68e804bd_isrForCrosstalkSources_1125900_35.stderr parsl.dataflow.dflow DEBUG: Task 3128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3128 has memoization hash 778216ca49644b71a18ab68c9a3cc48f parsl.dataflow.memoization INFO: Task 3128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ea700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3128 try 0 launched on executor multi with executor id 3129 parsl.dataflow.dflow INFO: Standard output for task 3128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/02b70dfa-b0ac-4d62-a336-3909eec1efa0_isrForCrosstalkSources_1125915_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/02b70dfa-b0ac-4d62-a336-3909eec1efa0_isrForCrosstalkSources_1125915_26.stderr parsl.dataflow.dflow DEBUG: Task 3129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3129 has memoization hash cc76648146f42e9744b41d7db1325c02 parsl.dataflow.memoization INFO: Task 3129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ea660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3129 try 0 launched on executor multi with executor id 3130 parsl.dataflow.dflow INFO: Standard output for task 3129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/b157fcfd-a27a-4196-9b5d-8ae66a95edd5_isrForCrosstalkSources_1125916_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/b157fcfd-a27a-4196-9b5d-8ae66a95edd5_isrForCrosstalkSources_1125916_49.stderr parsl.dataflow.dflow DEBUG: Task 3130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3130 has memoization hash f9c20b0f994e7fddc5c86711d4daf09d parsl.dataflow.memoization INFO: Task 3130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ea520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3130 try 0 launched on executor multi with executor id 3131 parsl.dataflow.dflow INFO: Standard output for task 3130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/0758c20f-15a6-4d49-b5ce-08083339506e_isrForCrosstalkSources_1125931_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/0758c20f-15a6-4d49-b5ce-08083339506e_isrForCrosstalkSources_1125931_29.stderr parsl.dataflow.dflow DEBUG: Task 3131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3131 has memoization hash 85be6b8efe0c3a3610e7e0357dffb02d parsl.dataflow.memoization INFO: Task 3131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ea5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3131 try 0 launched on executor multi with executor id 3132 parsl.dataflow.dflow INFO: Standard output for task 3131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/d4418d20-653b-48d4-b7e6-3fcd468433e5_isrForCrosstalkSources_1125913_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/d4418d20-653b-48d4-b7e6-3fcd468433e5_isrForCrosstalkSources_1125913_41.stderr parsl.dataflow.dflow DEBUG: Task 3132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3132 has memoization hash 6b29061f225f36fed72f90c851b27a39 parsl.dataflow.memoization INFO: Task 3132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ea480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3132 try 0 launched on executor multi with executor id 3133 parsl.dataflow.dflow INFO: Standard output for task 3132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c946a156-a327-42a4-801b-a8e3b87e656b_isrForCrosstalkSources_1125916_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c946a156-a327-42a4-801b-a8e3b87e656b_isrForCrosstalkSources_1125916_34.stderr parsl.dataflow.dflow DEBUG: Task 3133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3133 has memoization hash b2dcba3814e836cbf9e9c176270606f1 parsl.dataflow.memoization INFO: Task 3133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ea3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3133 try 0 launched on executor multi with executor id 3134 parsl.dataflow.dflow INFO: Standard output for task 3133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/5c5d4026-0108-40eb-8e81-81e0ceeb5352_isrForCrosstalkSources_1125945_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/5c5d4026-0108-40eb-8e81-81e0ceeb5352_isrForCrosstalkSources_1125945_59.stderr parsl.dataflow.dflow DEBUG: Task 3134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3134 has memoization hash b5b6e30da57650491235407553b8e0c0 parsl.dataflow.memoization INFO: Task 3134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ea340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3134 try 0 launched on executor multi with executor id 3135 parsl.dataflow.dflow INFO: Standard output for task 3134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/889fe82e-22fb-4258-a4a8-612900e7a7e0_isrForCrosstalkSources_1125926_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/889fe82e-22fb-4258-a4a8-612900e7a7e0_isrForCrosstalkSources_1125926_13.stderr parsl.dataflow.dflow DEBUG: Task 3135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3135 has memoization hash 22c66a2418a3f537619c23e9dd3a65b3 parsl.dataflow.memoization INFO: Task 3135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ea200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3135 try 0 launched on executor multi with executor id 3136 parsl.dataflow.dflow INFO: Standard output for task 3135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/2ab51468-9bdf-4f9e-9000-1f6d2bbeddc8_isrForCrosstalkSources_1125892_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/2ab51468-9bdf-4f9e-9000-1f6d2bbeddc8_isrForCrosstalkSources_1125892_42.stderr parsl.dataflow.dflow DEBUG: Task 3136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3136 has memoization hash a76f3943dd35c5ce1ee4ad8216423c6b parsl.dataflow.memoization INFO: Task 3136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ea2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3136 try 0 launched on executor multi with executor id 3137 parsl.dataflow.dflow INFO: Standard output for task 3136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a2f740b6-6775-43b3-8163-c0c426237d05_isrForCrosstalkSources_1125945_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a2f740b6-6775-43b3-8163-c0c426237d05_isrForCrosstalkSources_1125945_39.stderr parsl.dataflow.dflow DEBUG: Task 3137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3137 has memoization hash bffb0e0147c6c5c0457cebe6156de26f parsl.dataflow.memoization INFO: Task 3137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ea160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3137 try 0 launched on executor multi with executor id 3138 parsl.dataflow.dflow INFO: Standard output for task 3137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f7aae486-a69f-4184-9d42-3e9506416e65_isrForCrosstalkSources_1125939_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f7aae486-a69f-4184-9d42-3e9506416e65_isrForCrosstalkSources_1125939_6.stderr parsl.dataflow.dflow DEBUG: Task 3138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3138 has memoization hash a416477e7c904c773b0a09e2530c0622 parsl.dataflow.memoization INFO: Task 3138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ea0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:04 proc_lsst.multi:170 [INFO] [multi] job_id 16896435 proc_lsst.multi INFO: [multi] job_id 16896435 2024-03-07 15:07:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 15:07:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:04 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896435 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896435 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 16896435 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-07 15:07:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-07 15:07:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 15:07:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3138 try 0 launched on executor multi with executor id 3139 parsl.dataflow.dflow INFO: Standard output for task 3138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/78821392-951b-4d17-9ed7-fea41f6931f8_isrForCrosstalkSources_1125926_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/78821392-951b-4d17-9ed7-fea41f6931f8_isrForCrosstalkSources_1125926_29.stderr parsl.dataflow.dflow DEBUG: Task 3139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3139 has memoization hash b27809ec18be3773e923f575aaab3391 parsl.dataflow.memoization INFO: Task 3139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3139 try 0 launched on executor multi with executor id 3140 parsl.dataflow.dflow INFO: Standard output for task 3139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/a9564c54-8587-4b52-937f-04c12414e016_isrForCrosstalkSources_1125912_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/a9564c54-8587-4b52-937f-04c12414e016_isrForCrosstalkSources_1125912_38.stderr parsl.dataflow.dflow DEBUG: Task 3140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3140 has memoization hash e5ef6c03903651998480123581267b28 parsl.dataflow.memoization INFO: Task 3140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3140 try 0 launched on executor multi with executor id 3141 parsl.dataflow.dflow INFO: Standard output for task 3140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/13b7d01b-189b-4636-b501-d50b6724291c_isrForCrosstalkSources_1125935_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/13b7d01b-189b-4636-b501-d50b6724291c_isrForCrosstalkSources_1125935_17.stderr parsl.dataflow.dflow DEBUG: Task 3141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3141 has memoization hash cd717d1b3ada0fc81f0ee0ae92c5de1e parsl.dataflow.memoization INFO: Task 3141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ea020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3141 try 0 launched on executor multi with executor id 3142 parsl.dataflow.dflow INFO: Standard output for task 3141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/a0369935-1af2-411e-966b-db8f7e6931ba_isrForCrosstalkSources_1125922_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/a0369935-1af2-411e-966b-db8f7e6931ba_isrForCrosstalkSources_1125922_14.stderr parsl.dataflow.dflow DEBUG: Task 3142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3142 has memoization hash 3f33e2d7defe66815cd62c70da5cb528 parsl.dataflow.memoization INFO: Task 3142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3142 try 0 launched on executor multi with executor id 3143 parsl.dataflow.dflow INFO: Standard output for task 3142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/0d2a6833-9b37-408a-b192-09ca7a9f9680_isrForCrosstalkSources_1125929_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/0d2a6833-9b37-408a-b192-09ca7a9f9680_isrForCrosstalkSources_1125929_34.stderr parsl.dataflow.dflow DEBUG: Task 3143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3143 has memoization hash 63ef0e59d7888921ea8a255405caf1ac parsl.dataflow.memoization INFO: Task 3143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3143 try 0 launched on executor multi with executor id 3144 parsl.dataflow.dflow INFO: Standard output for task 3143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/a8143ab0-eb00-42e4-8a08-01f148c94388_isrForCrosstalkSources_1125926_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/a8143ab0-eb00-42e4-8a08-01f148c94388_isrForCrosstalkSources_1125926_10.stderr parsl.dataflow.dflow DEBUG: Task 3144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3144 has memoization hash f5e3dc84f3aabaa8da7e0e020874e0b8 parsl.dataflow.memoization INFO: Task 3144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3144 try 0 launched on executor multi with executor id 3145 parsl.dataflow.dflow INFO: Standard output for task 3144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/df683b0e-692e-4a43-86e8-c31a1d7e4884_isrForCrosstalkSources_1125918_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/df683b0e-692e-4a43-86e8-c31a1d7e4884_isrForCrosstalkSources_1125918_45.stderr parsl.dataflow.dflow DEBUG: Task 3145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3145 has memoization hash 753406a33bc8315e0f16039e3f602e14 parsl.dataflow.memoization INFO: Task 3145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3145 try 0 launched on executor multi with executor id 3146 parsl.dataflow.dflow INFO: Standard output for task 3145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/9e71dd50-d9a4-4dc9-80cc-ecf822acac69_isrForCrosstalkSources_1125944_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/9e71dd50-d9a4-4dc9-80cc-ecf822acac69_isrForCrosstalkSources_1125944_58.stderr parsl.dataflow.dflow DEBUG: Task 3146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3146 has memoization hash 65f5d488b17cff4cbe0384fcba71dbb1 parsl.dataflow.memoization INFO: Task 3146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3146 try 0 launched on executor multi with executor id 3147 parsl.dataflow.dflow INFO: Standard output for task 3146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/37ea9595-cda6-481e-8f4c-c01aead239db_isrForCrosstalkSources_1125892_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/37ea9595-cda6-481e-8f4c-c01aead239db_isrForCrosstalkSources_1125892_23.stderr parsl.dataflow.dflow DEBUG: Task 3147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3147 has memoization hash d91232f82f68c2a7f429a742953e840d parsl.dataflow.memoization INFO: Task 3147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3147 try 0 launched on executor multi with executor id 3148 parsl.dataflow.dflow INFO: Standard output for task 3147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/f0e2ce19-5beb-48ac-bef8-01eb9c905540_isrForCrosstalkSources_1125931_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/f0e2ce19-5beb-48ac-bef8-01eb9c905540_isrForCrosstalkSources_1125931_44.stderr parsl.dataflow.dflow DEBUG: Task 3148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3148 has memoization hash f05462fd1e0ac8236846f436d0f18c01 parsl.dataflow.memoization INFO: Task 3148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3148 try 0 launched on executor multi with executor id 3149 parsl.dataflow.dflow INFO: Standard output for task 3148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/3c752457-4fbd-4539-9a62-0f0f9e391107_isrForCrosstalkSources_1125946_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/3c752457-4fbd-4539-9a62-0f0f9e391107_isrForCrosstalkSources_1125946_18.stderr parsl.dataflow.dflow DEBUG: Task 3149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3149 has memoization hash 44a2712a8dc02626d8cc5553fb68d93f parsl.dataflow.memoization INFO: Task 3149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eb100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3149 try 0 launched on executor multi with executor id 3150 parsl.dataflow.dflow INFO: Standard output for task 3149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/6c5571ea-c485-400d-9aac-bf9bb3d1da5e_isrForCrosstalkSources_1125923_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/6c5571ea-c485-400d-9aac-bf9bb3d1da5e_isrForCrosstalkSources_1125923_44.stderr parsl.dataflow.dflow DEBUG: Task 3150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3150 has memoization hash ef2059854ab44a63d0b1ac7e7e8a3d56 parsl.dataflow.memoization INFO: Task 3150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3150 try 0 launched on executor multi with executor id 3151 parsl.dataflow.dflow INFO: Standard output for task 3150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/7e4f636f-6caa-4fcb-bef5-4bd082457c6b_isrForCrosstalkSources_1125932_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/7e4f636f-6caa-4fcb-bef5-4bd082457c6b_isrForCrosstalkSources_1125932_41.stderr parsl.dataflow.dflow DEBUG: Task 3151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3151 has memoization hash f2fec6e3f94973e66a7fcf5362e66313 parsl.dataflow.memoization INFO: Task 3151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ebf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3151 try 0 launched on executor multi with executor id 3152 parsl.dataflow.dflow INFO: Standard output for task 3151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2f250e38-f76e-43e0-b11f-1a68e3c8dbdb_isrForCrosstalkSources_1125898_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2f250e38-f76e-43e0-b11f-1a68e3c8dbdb_isrForCrosstalkSources_1125898_28.stderr parsl.dataflow.dflow DEBUG: Task 3152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3152 has memoization hash e562046357e341c5e4eb37dfa1e65551 parsl.dataflow.memoization INFO: Task 3152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393ebb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3152 try 0 launched on executor multi with executor id 3153 parsl.dataflow.dflow INFO: Standard output for task 3152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/7983e4c5-15ee-4a6f-aae7-dd8dab896752_isrForCrosstalkSources_1125893_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/7983e4c5-15ee-4a6f-aae7-dd8dab896752_isrForCrosstalkSources_1125893_25.stderr parsl.dataflow.dflow DEBUG: Task 3153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3153 has memoization hash 3fcb90e30588039c082f5a55414e50d4 parsl.dataflow.memoization INFO: Task 3153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3153 try 0 launched on executor multi with executor id 3154 parsl.dataflow.dflow INFO: Standard output for task 3153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/8c83a99d-1dcf-494a-924b-eddd6ea65078_isrForCrosstalkSources_1125902_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/8c83a99d-1dcf-494a-924b-eddd6ea65078_isrForCrosstalkSources_1125902_23.stderr parsl.dataflow.dflow DEBUG: Task 3154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3154 has memoization hash fb8f5239f1021f89285e9ba3f3ebaa25 parsl.dataflow.memoization INFO: Task 3154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3154 try 0 launched on executor multi with executor id 3155 parsl.dataflow.dflow INFO: Standard output for task 3154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/de9f5173-5cd9-480a-9b80-b50dd2113a64_isrForCrosstalkSources_1125888_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/de9f5173-5cd9-480a-9b80-b50dd2113a64_isrForCrosstalkSources_1125888_60.stderr parsl.dataflow.dflow DEBUG: Task 3155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3155 has memoization hash 9b47ba3cd31a7675e3893d6f037cd83a parsl.dataflow.memoization INFO: Task 3155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3155 try 0 launched on executor multi with executor id 3156 parsl.dataflow.dflow INFO: Standard output for task 3155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/eca60bcf-59e7-421b-bb64-ddf510d849d7_isrForCrosstalkSources_1125941_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/eca60bcf-59e7-421b-bb64-ddf510d849d7_isrForCrosstalkSources_1125941_36.stderr parsl.dataflow.dflow DEBUG: Task 3156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3156 has memoization hash 7322b9dbd17afafdb4ed112f084593a0 parsl.dataflow.memoization INFO: Task 3156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3156 try 0 launched on executor multi with executor id 3157 parsl.dataflow.dflow INFO: Standard output for task 3156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/1a5db8f0-9f63-45b4-8996-32ba332aad47_isrForCrosstalkSources_1125919_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/1a5db8f0-9f63-45b4-8996-32ba332aad47_isrForCrosstalkSources_1125919_12.stderr parsl.dataflow.dflow DEBUG: Task 3157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3157 has memoization hash 3ec97b829695e8b85c6de2b599fc794b parsl.dataflow.memoization INFO: Task 3157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3157 try 0 launched on executor multi with executor id 3158 parsl.dataflow.dflow INFO: Standard output for task 3157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/512764d4-ba7d-4547-8679-4e726d9d30a3_isrForCrosstalkSources_1125896_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/512764d4-ba7d-4547-8679-4e726d9d30a3_isrForCrosstalkSources_1125896_32.stderr parsl.dataflow.dflow DEBUG: Task 3158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3158 has memoization hash 925918ee39c769d9dbcdec5fc4501acd parsl.dataflow.memoization INFO: Task 3158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3158 try 0 launched on executor multi with executor id 3159 parsl.dataflow.dflow INFO: Standard output for task 3158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/94a0eeae-89ea-41bc-be78-ec3e45b88ce9_isrForCrosstalkSources_1125939_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/94a0eeae-89ea-41bc-be78-ec3e45b88ce9_isrForCrosstalkSources_1125939_20.stderr parsl.dataflow.dflow DEBUG: Task 3159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3159 has memoization hash 8b075b41d6c2503c12650eb3795c62dc parsl.dataflow.memoization INFO: Task 3159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3159 try 0 launched on executor multi with executor id 3160 parsl.dataflow.dflow INFO: Standard output for task 3159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/aae7912e-5bdd-4564-ba27-70f6882b8fa9_isrForCrosstalkSources_1125925_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/aae7912e-5bdd-4564-ba27-70f6882b8fa9_isrForCrosstalkSources_1125925_20.stderr parsl.dataflow.dflow DEBUG: Task 3160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3160 has memoization hash f93e44d90023425171618358aba33341 parsl.dataflow.memoization INFO: Task 3160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3160 try 0 launched on executor multi with executor id 3161 parsl.dataflow.dflow INFO: Standard output for task 3160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/31fc3117-7247-40ed-8e4e-7075f614b382_isrForCrosstalkSources_1125919_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/31fc3117-7247-40ed-8e4e-7075f614b382_isrForCrosstalkSources_1125919_13.stderr parsl.dataflow.dflow DEBUG: Task 3161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3161 has memoization hash ba28b9f55db922dd338e2cd286cf96c6 parsl.dataflow.memoization INFO: Task 3161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3161 try 0 launched on executor multi with executor id 3162 parsl.dataflow.dflow INFO: Standard output for task 3161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/cfa5d851-1d2f-4aaa-ba76-8a74c672c572_isrForCrosstalkSources_1125935_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/cfa5d851-1d2f-4aaa-ba76-8a74c672c572_isrForCrosstalkSources_1125935_40.stderr parsl.dataflow.dflow DEBUG: Task 3162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3162 has memoization hash 0f1375a286123304507e21329b896a79 parsl.dataflow.memoization INFO: Task 3162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:05 proc_lsst.multi:170 [INFO] [multi] job_id 16896437 proc_lsst.multi INFO: [multi] job_id 16896437 2024-03-07 15:07:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 15:07:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:05 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896437 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896437 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 16896437 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-07 15:07:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-07 15:07:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 15:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3162 try 0 launched on executor multi with executor id 3163 parsl.dataflow.dflow INFO: Standard output for task 3162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/db6d9fd3-4546-4bc4-a7e7-80a3f56ddca2_isrForCrosstalkSources_1125889_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/db6d9fd3-4546-4bc4-a7e7-80a3f56ddca2_isrForCrosstalkSources_1125889_30.stderr parsl.dataflow.dflow DEBUG: Task 3163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3163 has memoization hash ffe7426da8dc5fc6faddac356f18d09c parsl.dataflow.memoization INFO: Task 3163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3163 try 0 launched on executor multi with executor id 3164 parsl.dataflow.dflow INFO: Standard output for task 3163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/706d7fcb-d21b-4b03-a387-a7a49d0702ca_isrForCrosstalkSources_1125942_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/706d7fcb-d21b-4b03-a387-a7a49d0702ca_isrForCrosstalkSources_1125942_14.stderr parsl.dataflow.dflow DEBUG: Task 3164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3164 has memoization hash a52a7b47df78d219a28f8a2bd20a59c0 parsl.dataflow.memoization INFO: Task 3164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3164 try 0 launched on executor multi with executor id 3165 parsl.dataflow.dflow INFO: Standard output for task 3164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/e5ccbdd7-5272-4575-9771-e20fec97cf1f_isrForCrosstalkSources_1125938_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/e5ccbdd7-5272-4575-9771-e20fec97cf1f_isrForCrosstalkSources_1125938_14.stderr parsl.dataflow.dflow DEBUG: Task 3165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3165 has memoization hash 5149952dce08900cda9706da8df8f7b2 parsl.dataflow.memoization INFO: Task 3165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3165 try 0 launched on executor multi with executor id 3166 parsl.dataflow.dflow INFO: Standard output for task 3165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/6e1b1fe0-c8de-4830-b2af-af0cd9c17965_isrForCrosstalkSources_1125924_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/6e1b1fe0-c8de-4830-b2af-af0cd9c17965_isrForCrosstalkSources_1125924_34.stderr parsl.dataflow.dflow DEBUG: Task 3166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3166 has memoization hash 5a165a74944a0458106c54b0ff211a5d parsl.dataflow.memoization INFO: Task 3166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3166 try 0 launched on executor multi with executor id 3167 parsl.dataflow.dflow INFO: Standard output for task 3166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/0c8a54cb-dc41-401f-a66f-0545bc8a7a06_isrForCrosstalkSources_1125936_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/0c8a54cb-dc41-401f-a66f-0545bc8a7a06_isrForCrosstalkSources_1125936_5.stderr parsl.dataflow.dflow DEBUG: Task 3167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3167 has memoization hash 0ba808df2fb26a35ca3224288cca203d parsl.dataflow.memoization INFO: Task 3167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392872e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3167 try 0 launched on executor multi with executor id 3168 parsl.dataflow.dflow INFO: Standard output for task 3167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/c349fd1d-1938-4049-9d52-d1c36a57788b_isrForCrosstalkSources_1125908_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/c349fd1d-1938-4049-9d52-d1c36a57788b_isrForCrosstalkSources_1125908_29.stderr parsl.dataflow.dflow DEBUG: Task 3168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3168 has memoization hash b89046d9b6ae77ffde112047edbaad3d parsl.dataflow.memoization INFO: Task 3168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3168 try 0 launched on executor multi with executor id 3169 parsl.dataflow.dflow INFO: Standard output for task 3168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/aa4c5be3-7bab-41fb-ade0-270f29312b00_isrForCrosstalkSources_1125926_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/aa4c5be3-7bab-41fb-ade0-270f29312b00_isrForCrosstalkSources_1125926_30.stderr parsl.dataflow.dflow DEBUG: Task 3169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3169 has memoization hash 039e7bc809a9acb18828085ae7de5fef parsl.dataflow.memoization INFO: Task 3169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392840e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3169 try 0 launched on executor multi with executor id 3170 parsl.dataflow.dflow INFO: Standard output for task 3169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/7548874f-429e-4b52-87dc-ea3409b9c059_isrForCrosstalkSources_1125903_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/7548874f-429e-4b52-87dc-ea3409b9c059_isrForCrosstalkSources_1125903_6.stderr parsl.dataflow.dflow DEBUG: Task 3170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3170 has memoization hash e195a147064e7b6d11d8fde7c562fa61 parsl.dataflow.memoization INFO: Task 3170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3170 try 0 launched on executor multi with executor id 3171 parsl.dataflow.dflow INFO: Standard output for task 3170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/8e379050-1148-4c8b-a2be-dbb6a0e3368b_isrForCrosstalkSources_1125888_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/8e379050-1148-4c8b-a2be-dbb6a0e3368b_isrForCrosstalkSources_1125888_54.stderr parsl.dataflow.dflow DEBUG: Task 3171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3171 has memoization hash 98e00efdb3c4370ab8a7dfc82896df0a parsl.dataflow.memoization INFO: Task 3171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3171 try 0 launched on executor multi with executor id 3172 parsl.dataflow.dflow INFO: Standard output for task 3171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/adcb5d9e-0da4-454a-b5ca-1b13a0a1908b_isrForCrosstalkSources_1125938_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/adcb5d9e-0da4-454a-b5ca-1b13a0a1908b_isrForCrosstalkSources_1125938_28.stderr parsl.dataflow.dflow DEBUG: Task 3172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3172 has memoization hash c02f972eaf45a2059970bb8c0548c10c parsl.dataflow.memoization INFO: Task 3172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3172 try 0 launched on executor multi with executor id 3173 parsl.dataflow.dflow INFO: Standard output for task 3172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/9d3db9d0-b9b3-4f2a-bd07-344547ab2191_isrForCrosstalkSources_1125941_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/9d3db9d0-b9b3-4f2a-bd07-344547ab2191_isrForCrosstalkSources_1125941_22.stderr parsl.dataflow.dflow DEBUG: Task 3173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3173 has memoization hash 32bc4b67ebe87f2431ef5d30a6116964 parsl.dataflow.memoization INFO: Task 3173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392842c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3173 try 0 launched on executor multi with executor id 3174 parsl.dataflow.dflow INFO: Standard output for task 3173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/98e22b85-600b-4628-866b-8eb7176d49a4_isrForCrosstalkSources_1125901_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/98e22b85-600b-4628-866b-8eb7176d49a4_isrForCrosstalkSources_1125901_14.stderr parsl.dataflow.dflow DEBUG: Task 3174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3174 has memoization hash 80ff6edf8e3a4cc932ab95da620ff004 parsl.dataflow.memoization INFO: Task 3174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3174 try 0 launched on executor multi with executor id 3175 parsl.dataflow.dflow INFO: Standard output for task 3174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/6530a0cf-131f-488a-b254-6e1917c2f6a7_isrForCrosstalkSources_1125901_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/6530a0cf-131f-488a-b254-6e1917c2f6a7_isrForCrosstalkSources_1125901_17.stderr parsl.dataflow.dflow DEBUG: Task 3175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3175 has memoization hash 3d4ff28420521cc5a49ae582ef7c90eb parsl.dataflow.memoization INFO: Task 3175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3175 try 0 launched on executor multi with executor id 3176 parsl.dataflow.dflow INFO: Standard output for task 3175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/23bba5a2-fbcd-41c6-96f7-3c526e448cd9_isrForCrosstalkSources_1125906_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/23bba5a2-fbcd-41c6-96f7-3c526e448cd9_isrForCrosstalkSources_1125906_9.stderr parsl.dataflow.dflow DEBUG: Task 3176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3176 has memoization hash 1640c337e5f191f0d228a90bd55d8169 parsl.dataflow.memoization INFO: Task 3176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392844a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3176 try 0 launched on executor multi with executor id 3177 parsl.dataflow.dflow INFO: Standard output for task 3176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e0dd845f-2d89-4f15-8a4b-16cf29cba232_isrForCrosstalkSources_1125916_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e0dd845f-2d89-4f15-8a4b-16cf29cba232_isrForCrosstalkSources_1125916_50.stderr parsl.dataflow.dflow DEBUG: Task 3177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3177 has memoization hash b3b1cf2fe8d138dc5edb4eee1c184726 parsl.dataflow.memoization INFO: Task 3177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3177 try 0 launched on executor multi with executor id 3178 parsl.dataflow.dflow INFO: Standard output for task 3177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/482f6b11-1163-4135-ae88-3eb4f2f6115f_isrForCrosstalkSources_1125903_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/482f6b11-1163-4135-ae88-3eb4f2f6115f_isrForCrosstalkSources_1125903_54.stderr parsl.dataflow.dflow DEBUG: Task 3178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3178 has memoization hash 7063d2feb01a8ff792a5e58b3fa8d23f parsl.dataflow.memoization INFO: Task 3178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3178 try 0 launched on executor multi with executor id 3179 parsl.dataflow.dflow INFO: Standard output for task 3178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/33504d93-8ac3-4faf-ac12-f716fdcfdd95_isrForCrosstalkSources_1125905_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/33504d93-8ac3-4faf-ac12-f716fdcfdd95_isrForCrosstalkSources_1125905_46.stderr parsl.dataflow.dflow DEBUG: Task 3179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3179 has memoization hash ee928d189263e971a28e22c26a837a07 parsl.dataflow.memoization INFO: Task 3179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3179 try 0 launched on executor multi with executor id 3180 parsl.dataflow.dflow INFO: Standard output for task 3179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/50ee72f6-f2d9-4c03-8001-25dcb9c5daf4_isrForCrosstalkSources_1125907_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/50ee72f6-f2d9-4c03-8001-25dcb9c5daf4_isrForCrosstalkSources_1125907_55.stderr parsl.dataflow.dflow DEBUG: Task 3180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3180 has memoization hash 8d0748838b19078ba19043fd31b320f7 parsl.dataflow.memoization INFO: Task 3180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392845e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3180 try 0 launched on executor multi with executor id 3181 parsl.dataflow.dflow INFO: Standard output for task 3180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/b3af87a5-49f3-4ae5-b63f-3bc9904c83cd_isrForCrosstalkSources_1125916_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/b3af87a5-49f3-4ae5-b63f-3bc9904c83cd_isrForCrosstalkSources_1125916_54.stderr parsl.dataflow.dflow DEBUG: Task 3181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3181 has memoization hash 2685348201d672f0cb1470dcd27ad9a9 parsl.dataflow.memoization INFO: Task 3181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392847c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3181 try 0 launched on executor multi with executor id 3182 parsl.dataflow.dflow INFO: Standard output for task 3181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/9f59011d-8e10-4722-aa5a-f3dbeee381eb_isrForCrosstalkSources_1125912_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/9f59011d-8e10-4722-aa5a-f3dbeee381eb_isrForCrosstalkSources_1125912_22.stderr parsl.dataflow.dflow DEBUG: Task 3182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3182 has memoization hash ee5b320007bb6e0156bc46f9424cba34 parsl.dataflow.memoization INFO: Task 3182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3182 try 0 launched on executor multi with executor id 3183 parsl.dataflow.dflow INFO: Standard output for task 3182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/5b92f826-772d-4c39-81bf-27612ec51945_isrForCrosstalkSources_1125943_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/5b92f826-772d-4c39-81bf-27612ec51945_isrForCrosstalkSources_1125943_29.stderr parsl.dataflow.dflow DEBUG: Task 3183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3183 has memoization hash 459f2d21ca6bbd6c4d3eec1e5b3f4fef parsl.dataflow.memoization INFO: Task 3183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3183 try 0 launched on executor multi with executor id 3184 parsl.dataflow.dflow INFO: Standard output for task 3183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/90ad9681-1774-452f-afcc-599bb51ab18a_isrForCrosstalkSources_1125893_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/90ad9681-1774-452f-afcc-599bb51ab18a_isrForCrosstalkSources_1125893_59.stderr parsl.dataflow.dflow DEBUG: Task 3184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3184 has memoization hash 9be409203ba0465a1471755c60a94e2d parsl.dataflow.memoization INFO: Task 3184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:05 proc_lsst.multi:170 [INFO] [multi] job_id 16896438 proc_lsst.multi INFO: [multi] job_id 16896438 2024-03-07 15:07:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 15:07:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:05 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896438 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896438 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 16896438 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-07 15:07:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-07 15:07:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 15:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3184 try 0 launched on executor multi with executor id 3185 parsl.dataflow.dflow INFO: Standard output for task 3184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/1b4a3194-f411-4514-ad25-e3b41deaf281_isrForCrosstalkSources_1125919_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/1b4a3194-f411-4514-ad25-e3b41deaf281_isrForCrosstalkSources_1125919_49.stderr parsl.dataflow.dflow DEBUG: Task 3185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3185 has memoization hash 28812615dec67fc78e993674abda6aa2 parsl.dataflow.memoization INFO: Task 3185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3185 try 0 launched on executor multi with executor id 3186 parsl.dataflow.dflow INFO: Standard output for task 3185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/9461ed30-7de8-4cab-9a18-4ae5f90a75ac_isrForCrosstalkSources_1125919_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/9461ed30-7de8-4cab-9a18-4ae5f90a75ac_isrForCrosstalkSources_1125919_23.stderr parsl.dataflow.dflow DEBUG: Task 3186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3186 has memoization hash 5a7aae04166ae3305e57ceeb303924fd parsl.dataflow.memoization INFO: Task 3186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393eb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3186 try 0 launched on executor multi with executor id 3187 parsl.dataflow.dflow INFO: Standard output for task 3186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/17f28ba7-c350-4b88-b90b-a66a5545bee2_isrForCrosstalkSources_1125896_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/17f28ba7-c350-4b88-b90b-a66a5545bee2_isrForCrosstalkSources_1125896_54.stderr parsl.dataflow.dflow DEBUG: Task 3187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3187 has memoization hash 4017b16578b2129787b35040901aa06d parsl.dataflow.memoization INFO: Task 3187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a393e9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3187 try 0 launched on executor multi with executor id 3188 parsl.dataflow.dflow INFO: Standard output for task 3187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/b827aad5-981a-469b-bf12-15fe55efa624_isrForCrosstalkSources_1125901_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/b827aad5-981a-469b-bf12-15fe55efa624_isrForCrosstalkSources_1125901_47.stderr parsl.dataflow.dflow DEBUG: Task 3188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3188 has memoization hash 44c96ba91b4315a3ee37e9bca5aa5313 parsl.dataflow.memoization INFO: Task 3188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3188 try 0 launched on executor multi with executor id 3189 parsl.dataflow.dflow INFO: Standard output for task 3188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/055a1d81-8f3c-4a3c-a4fe-e6599727a94a_isrForCrosstalkSources_1125929_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/055a1d81-8f3c-4a3c-a4fe-e6599727a94a_isrForCrosstalkSources_1125929_38.stderr parsl.dataflow.dflow DEBUG: Task 3189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3189 has memoization hash 5754f7dcc71a7ec19f9b597fd69d51af parsl.dataflow.memoization INFO: Task 3189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3189 try 0 launched on executor multi with executor id 3190 parsl.dataflow.dflow INFO: Standard output for task 3189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/103bae7b-67d9-4b4b-b39c-c924ceb9c0d1_isrForCrosstalkSources_1125938_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/103bae7b-67d9-4b4b-b39c-c924ceb9c0d1_isrForCrosstalkSources_1125938_9.stderr parsl.dataflow.dflow DEBUG: Task 3190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3190 has memoization hash d48a4328b58b81b4d492bbc42d08ca75 parsl.dataflow.memoization INFO: Task 3190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392849a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3190 try 0 launched on executor multi with executor id 3191 parsl.dataflow.dflow INFO: Standard output for task 3190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d440bd6f-5cfb-4a02-b2a4-7ccf89646aeb_isrForCrosstalkSources_1125911_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d440bd6f-5cfb-4a02-b2a4-7ccf89646aeb_isrForCrosstalkSources_1125911_62.stderr parsl.dataflow.dflow DEBUG: Task 3191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3191 has memoization hash 8b654307219d51119ceebf38409d6513 parsl.dataflow.memoization INFO: Task 3191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3191 try 0 launched on executor multi with executor id 3192 parsl.dataflow.dflow INFO: Standard output for task 3191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/7ba76eca-3d33-4155-8c35-9d1276dd106b_isrForCrosstalkSources_1125922_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/7ba76eca-3d33-4155-8c35-9d1276dd106b_isrForCrosstalkSources_1125922_31.stderr parsl.dataflow.dflow DEBUG: Task 3192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3192 has memoization hash b6f6528a6cfe75d11c4e6658ed17c62a parsl.dataflow.memoization INFO: Task 3192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3192 try 0 launched on executor multi with executor id 3193 parsl.dataflow.dflow INFO: Standard output for task 3192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/612a85d0-c8e5-4143-8a89-912fac6e1105_isrForCrosstalkSources_1125926_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/612a85d0-c8e5-4143-8a89-912fac6e1105_isrForCrosstalkSources_1125926_2.stderr parsl.dataflow.dflow DEBUG: Task 3193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3193 has memoization hash f4153bb8fe90d91937c438a2ba6050e6 parsl.dataflow.memoization INFO: Task 3193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3193 try 0 launched on executor multi with executor id 3194 parsl.dataflow.dflow INFO: Standard output for task 3193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/2c7058a4-925a-45c5-83e2-643dd2e06d03_isrForCrosstalkSources_1125920_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/2c7058a4-925a-45c5-83e2-643dd2e06d03_isrForCrosstalkSources_1125920_43.stderr parsl.dataflow.dflow DEBUG: Task 3194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3194 has memoization hash 6a04a4eb3278449505b518d0ec2862e6 parsl.dataflow.memoization INFO: Task 3194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3194 try 0 launched on executor multi with executor id 3195 parsl.dataflow.dflow INFO: Standard output for task 3194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/ac61d890-8f07-4ec2-8c97-f43803ea04fb_isrForCrosstalkSources_1125933_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/ac61d890-8f07-4ec2-8c97-f43803ea04fb_isrForCrosstalkSources_1125933_29.stderr parsl.dataflow.dflow DEBUG: Task 3195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3195 has memoization hash f1be206e5171e9fbe9cd7dee9c071db1 parsl.dataflow.memoization INFO: Task 3195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3195 try 0 launched on executor multi with executor id 3196 parsl.dataflow.dflow INFO: Standard output for task 3195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/fe65b84d-bb08-420f-8580-3eaeb59e4a81_isrForCrosstalkSources_1125939_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/fe65b84d-bb08-420f-8580-3eaeb59e4a81_isrForCrosstalkSources_1125939_4.stderr parsl.dataflow.dflow DEBUG: Task 3196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3196 has memoization hash 837d71b83f50c9e9eaf80137dceecdfc parsl.dataflow.memoization INFO: Task 3196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3196 try 0 launched on executor multi with executor id 3197 parsl.dataflow.dflow INFO: Standard output for task 3196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/8a4a6994-aa58-44ae-a7a9-6995def2f5a6_isrForCrosstalkSources_1125890_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/8a4a6994-aa58-44ae-a7a9-6995def2f5a6_isrForCrosstalkSources_1125890_28.stderr parsl.dataflow.dflow DEBUG: Task 3197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3197 has memoization hash 5f337d777b700c909afdc70b5d28448a parsl.dataflow.memoization INFO: Task 3197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3197 try 0 launched on executor multi with executor id 3198 parsl.dataflow.dflow INFO: Standard output for task 3197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/88a4b89c-adb3-4bf2-bf7a-74f8fd11bb8c_isrForCrosstalkSources_1125944_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/88a4b89c-adb3-4bf2-bf7a-74f8fd11bb8c_isrForCrosstalkSources_1125944_13.stderr parsl.dataflow.dflow DEBUG: Task 3198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3198 has memoization hash 5e511434977dc335d4bd07c65abd31d3 parsl.dataflow.memoization INFO: Task 3198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3198 try 0 launched on executor multi with executor id 3199 parsl.dataflow.dflow INFO: Standard output for task 3198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/31f4b82e-6fe1-4d67-a8a5-ba4f70bc2ffe_isrForCrosstalkSources_1125914_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/31f4b82e-6fe1-4d67-a8a5-ba4f70bc2ffe_isrForCrosstalkSources_1125914_42.stderr parsl.dataflow.dflow DEBUG: Task 3199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3199 has memoization hash 8ddffe05f6cfa13b528d34e68f502abf parsl.dataflow.memoization INFO: Task 3199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392851c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3199 try 0 launched on executor multi with executor id 3200 parsl.dataflow.dflow INFO: Standard output for task 3199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/076008b4-c829-429c-9ec6-88f0a215c663_isrForCrosstalkSources_1125920_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/076008b4-c829-429c-9ec6-88f0a215c663_isrForCrosstalkSources_1125920_33.stderr parsl.dataflow.dflow DEBUG: Task 3200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3200 has memoization hash 3605ac668728013941d871270d25fbb1 parsl.dataflow.memoization INFO: Task 3200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3200 try 0 launched on executor multi with executor id 3201 parsl.dataflow.dflow INFO: Standard output for task 3200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/09a0aee3-3cfc-4e3b-ae30-628a4872896b_isrForCrosstalkSources_1125931_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/09a0aee3-3cfc-4e3b-ae30-628a4872896b_isrForCrosstalkSources_1125931_15.stderr parsl.dataflow.dflow DEBUG: Task 3201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3201 has memoization hash a8e97aec7c606f73391d6162f598a1ea parsl.dataflow.memoization INFO: Task 3201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3201 try 0 launched on executor multi with executor id 3202 parsl.dataflow.dflow INFO: Standard output for task 3201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d18451c3-7914-4978-9154-f3104ff82f52_isrForCrosstalkSources_1125911_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d18451c3-7914-4978-9154-f3104ff82f52_isrForCrosstalkSources_1125911_13.stderr parsl.dataflow.dflow DEBUG: Task 3202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3202 has memoization hash 38a9956c7fc526552a7d01369df7958e parsl.dataflow.memoization INFO: Task 3202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3202 try 0 launched on executor multi with executor id 3203 parsl.dataflow.dflow INFO: Standard output for task 3202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/67022975-7b1f-46a7-9a71-f020e7fb25f9_isrForCrosstalkSources_1125892_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/67022975-7b1f-46a7-9a71-f020e7fb25f9_isrForCrosstalkSources_1125892_48.stderr parsl.dataflow.dflow DEBUG: Task 3203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3203 has memoization hash f6c28905fbf0bb3e65e8ea84dbe242e9 parsl.dataflow.memoization INFO: Task 3203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3203 try 0 launched on executor multi with executor id 3204 parsl.dataflow.dflow INFO: Standard output for task 3203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/1d12fe05-7473-4835-aa8c-1e6cd8c8999b_isrForCrosstalkSources_1125928_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/1d12fe05-7473-4835-aa8c-1e6cd8c8999b_isrForCrosstalkSources_1125928_1.stderr parsl.dataflow.dflow DEBUG: Task 3204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3204 has memoization hash 78498ea8ab4ead14004677e3a0ff1e96 parsl.dataflow.memoization INFO: Task 3204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392854e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3204 try 0 launched on executor multi with executor id 3205 parsl.dataflow.dflow INFO: Standard output for task 3204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/aa5d010b-4b69-4bd6-9779-947b090cbc74_isrForCrosstalkSources_1125916_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/aa5d010b-4b69-4bd6-9779-947b090cbc74_isrForCrosstalkSources_1125916_44.stderr parsl.dataflow.dflow DEBUG: Task 3205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3205 has memoization hash 658f1f2cf84106a58aaf00fe5cf766e6 parsl.dataflow.memoization INFO: Task 3205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3205 try 0 launched on executor multi with executor id 3206 parsl.dataflow.dflow INFO: Standard output for task 3205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/9ff4a26a-ebf2-4695-a4df-2105d4a7467f_isrForCrosstalkSources_1125947_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/9ff4a26a-ebf2-4695-a4df-2105d4a7467f_isrForCrosstalkSources_1125947_61.stderr parsl.dataflow.dflow DEBUG: Task 3206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3206 has memoization hash e861ff95d17a588981a31f0b9648c87a parsl.dataflow.memoization INFO: Task 3206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392853a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3206 try 0 launched on executor multi with executor id 3207 parsl.dataflow.dflow INFO: Standard output for task 3206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/914e2c46-130c-49f4-acf5-3ea2fe0b4b8c_isrForCrosstalkSources_1125937_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/914e2c46-130c-49f4-acf5-3ea2fe0b4b8c_isrForCrosstalkSources_1125937_45.stderr parsl.dataflow.dflow DEBUG: Task 3207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3207 has memoization hash ac9e917023d7a2a71fe00c317d1fe121 parsl.dataflow.memoization INFO: Task 3207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3207 try 0 launched on executor multi with executor id 3208 parsl.dataflow.dflow INFO: Standard output for task 3207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/92fef1c3-3012-4035-9868-8b9b35aa9986_isrForCrosstalkSources_1125925_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/92fef1c3-3012-4035-9868-8b9b35aa9986_isrForCrosstalkSources_1125925_49.stderr parsl.dataflow.dflow DEBUG: Task 3208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3208 has memoization hash cd74d07176dd0ee60fbf4cfef1b33635 parsl.dataflow.memoization INFO: Task 3208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3208 try 0 launched on executor multi with executor id 3209 parsl.dataflow.dflow INFO: Standard output for task 3208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/d2831846-4a2b-48ea-be77-60f48fb193ce_isrForCrosstalkSources_1125925_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/d2831846-4a2b-48ea-be77-60f48fb193ce_isrForCrosstalkSources_1125925_53.stderr parsl.dataflow.dflow DEBUG: Task 3209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3209 has memoization hash e767fce6d302d21ebd3a23ade34acb59 parsl.dataflow.memoization INFO: Task 3209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3209 try 0 launched on executor multi with executor id 3210 parsl.dataflow.dflow INFO: Standard output for task 3209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/1b03f3d4-6e64-4da0-b33e-f9d6c6068e50_isrForCrosstalkSources_1125897_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/1b03f3d4-6e64-4da0-b33e-f9d6c6068e50_isrForCrosstalkSources_1125897_60.stderr parsl.dataflow.dflow DEBUG: Task 3210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3210 has memoization hash 4657e1b6c6e1961de3fb8f39c7a3ccc6 parsl.dataflow.memoization INFO: Task 3210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392856c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3210 try 0 launched on executor multi with executor id 3211 parsl.dataflow.dflow INFO: Standard output for task 3210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/4c964d66-0d75-464a-ba94-9d375aa0e8f5_isrForCrosstalkSources_1125926_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/4c964d66-0d75-464a-ba94-9d375aa0e8f5_isrForCrosstalkSources_1125926_34.stderr parsl.dataflow.dflow DEBUG: Task 3211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3211 has memoization hash def70b29989b227e79d15e1e847b1cf7 parsl.dataflow.memoization INFO: Task 3211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3211 try 0 launched on executor multi with executor id 3212 parsl.dataflow.dflow INFO: Standard output for task 3211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/185286ee-bb96-4995-8efd-ee3f41d32eab_isrForCrosstalkSources_1125926_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/185286ee-bb96-4995-8efd-ee3f41d32eab_isrForCrosstalkSources_1125926_3.stderr parsl.dataflow.dflow DEBUG: Task 3212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3212 has memoization hash 81229536f94cb03241b56c112eed5d2e parsl.dataflow.memoization INFO: Task 3212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3212 try 0 launched on executor multi with executor id 3213 parsl.dataflow.dflow INFO: Standard output for task 3212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/95aee4d9-145f-4e2d-bc18-5c53aeac29cc_isrForCrosstalkSources_1125919_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/95aee4d9-145f-4e2d-bc18-5c53aeac29cc_isrForCrosstalkSources_1125919_53.stderr parsl.dataflow.dflow DEBUG: Task 3213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3213 has memoization hash 5f0ca6cedf066353037b431a138a61a4 parsl.dataflow.memoization INFO: Task 3213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392859e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3213 try 0 launched on executor multi with executor id 3214 parsl.dataflow.dflow INFO: Standard output for task 3213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/b59242c7-f11d-4b71-bbd3-a540fb055fa6_isrForCrosstalkSources_1125905_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/b59242c7-f11d-4b71-bbd3-a540fb055fa6_isrForCrosstalkSources_1125905_39.stderr parsl.dataflow.dflow DEBUG: Task 3214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3214 has memoization hash 839a45282c5cea98d351c59904b23a4a parsl.dataflow.memoization INFO: Task 3214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3214 try 0 launched on executor multi with executor id 3215 parsl.dataflow.dflow INFO: Standard output for task 3214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/1a5754ac-8687-4531-9399-5d47c40551cf_isrForCrosstalkSources_1125923_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/1a5754ac-8687-4531-9399-5d47c40551cf_isrForCrosstalkSources_1125923_22.stderr parsl.dataflow.dflow DEBUG: Task 3215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3215 has memoization hash 88bcea58280efb7f744c89a5a3a1d7cc parsl.dataflow.memoization INFO: Task 3215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3215 try 0 launched on executor multi with executor id 3216 parsl.dataflow.dflow INFO: Standard output for task 3215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/00214485-930f-4b12-a44a-d60cc48ae2ce_isrForCrosstalkSources_1125930_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/00214485-930f-4b12-a44a-d60cc48ae2ce_isrForCrosstalkSources_1125930_20.stderr parsl.dataflow.dflow DEBUG: Task 3216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3216 has memoization hash 42217e1b45afa695754ac6c3cea1f05a parsl.dataflow.memoization INFO: Task 3216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3216 try 0 launched on executor multi with executor id 3217 parsl.dataflow.dflow INFO: Standard output for task 3216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/4e88c3c2-1d5a-4f6e-baf0-956445eddfde_isrForCrosstalkSources_1125903_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/4e88c3c2-1d5a-4f6e-baf0-956445eddfde_isrForCrosstalkSources_1125903_13.stderr parsl.dataflow.dflow DEBUG: Task 3217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3217 has memoization hash ee4bad369d1864e481ee3cb065c49486 parsl.dataflow.memoization INFO: Task 3217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392858a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3217 try 0 launched on executor multi with executor id 3218 parsl.dataflow.dflow INFO: Standard output for task 3217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/33162fba-68b1-4e13-967a-c53f29634ac7_isrForCrosstalkSources_1125939_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/33162fba-68b1-4e13-967a-c53f29634ac7_isrForCrosstalkSources_1125939_21.stderr parsl.dataflow.dflow DEBUG: Task 3218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3218 has memoization hash b4e1b8f6f6b5df69732ea8053b44e120 parsl.dataflow.memoization INFO: Task 3218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3218 try 0 launched on executor multi with executor id 3219 parsl.dataflow.dflow INFO: Standard output for task 3218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/452bccb1-717b-46b2-8698-bde8847ae5cf_isrForCrosstalkSources_1125930_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/452bccb1-717b-46b2-8698-bde8847ae5cf_isrForCrosstalkSources_1125930_36.stderr parsl.dataflow.dflow DEBUG: Task 3219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3219 has memoization hash 8d8c2b9c0e16a7fef06f3c9b361fb1cf parsl.dataflow.memoization INFO: Task 3219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3219 try 0 launched on executor multi with executor id 3220 parsl.dataflow.dflow INFO: Standard output for task 3219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/e4a00982-56b2-4bff-ac60-ed615dfb7b40_isrForCrosstalkSources_1125890_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/e4a00982-56b2-4bff-ac60-ed615dfb7b40_isrForCrosstalkSources_1125890_15.stderr parsl.dataflow.dflow DEBUG: Task 3220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3220 has memoization hash ec93581c5ae18f42c8b904ca4dd929dc parsl.dataflow.memoization INFO: Task 3220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:05 proc_lsst.multi:170 [INFO] [multi] job_id 16896439 proc_lsst.multi INFO: [multi] job_id 16896439 2024-03-07 15:07:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 15:07:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:05 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896439 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896439 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 16896439 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-07 15:07:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-07 15:07:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 15:07:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3220 try 0 launched on executor multi with executor id 3221 parsl.dataflow.dflow INFO: Standard output for task 3220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/ef433f7e-a37e-4b40-a404-8c395df9a137_isrForCrosstalkSources_1125895_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/ef433f7e-a37e-4b40-a404-8c395df9a137_isrForCrosstalkSources_1125895_27.stderr parsl.dataflow.dflow DEBUG: Task 3221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3221 has memoization hash cbed9c94612e7c00cdb95c66ba261249 parsl.dataflow.memoization INFO: Task 3221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3221 try 0 launched on executor multi with executor id 3222 parsl.dataflow.dflow INFO: Standard output for task 3221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/85ee3770-afd2-4252-a08d-b1f458d028c6_isrForCrosstalkSources_1125934_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/85ee3770-afd2-4252-a08d-b1f458d028c6_isrForCrosstalkSources_1125934_35.stderr parsl.dataflow.dflow DEBUG: Task 3222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3222 has memoization hash 977f8e31af44f5ea89348e283961c7ce parsl.dataflow.memoization INFO: Task 3222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3222 try 0 launched on executor multi with executor id 3223 parsl.dataflow.dflow INFO: Standard output for task 3222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/9dda99eb-3446-4185-91a0-7bf582966f2b_isrForCrosstalkSources_1125930_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/9dda99eb-3446-4185-91a0-7bf582966f2b_isrForCrosstalkSources_1125930_45.stderr parsl.dataflow.dflow DEBUG: Task 3223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3223 has memoization hash f89db9a92333b0641ad520f6a0c77dde parsl.dataflow.memoization INFO: Task 3223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392871a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3223 try 0 launched on executor multi with executor id 3224 parsl.dataflow.dflow INFO: Standard output for task 3223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a3a0ac6b-7f18-47d1-bed5-d0b97cd95ccd_isrForCrosstalkSources_1125921_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a3a0ac6b-7f18-47d1-bed5-d0b97cd95ccd_isrForCrosstalkSources_1125921_59.stderr parsl.dataflow.dflow DEBUG: Task 3224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3224 has memoization hash 1741fb841037426b301b69a286afce90 parsl.dataflow.memoization INFO: Task 3224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39363f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3224 try 0 launched on executor multi with executor id 3225 parsl.dataflow.dflow INFO: Standard output for task 3224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c67a3946-44d9-4fe5-a85c-b5947268dc26_isrForCrosstalkSources_1125910_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c67a3946-44d9-4fe5-a85c-b5947268dc26_isrForCrosstalkSources_1125910_38.stderr parsl.dataflow.dflow DEBUG: Task 3225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3225 has memoization hash 2fcac2580eac7f0d4a76e103ecadb4d7 parsl.dataflow.memoization INFO: Task 3225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3225 try 0 launched on executor multi with executor id 3226 parsl.dataflow.dflow INFO: Standard output for task 3225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c89f5f34-3d24-4d1e-af47-f956ef18b25e_isrForCrosstalkSources_1125904_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c89f5f34-3d24-4d1e-af47-f956ef18b25e_isrForCrosstalkSources_1125904_54.stderr parsl.dataflow.dflow DEBUG: Task 3226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3226 has memoization hash fc314f4cee646baf1ab6cc10294df1fe parsl.dataflow.memoization INFO: Task 3226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ed8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3226 try 0 launched on executor multi with executor id 3227 parsl.dataflow.dflow INFO: Standard output for task 3226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4121790e-2e67-44f6-bd17-c1f34e0713c1_isrForCrosstalkSources_1125893_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4121790e-2e67-44f6-bd17-c1f34e0713c1_isrForCrosstalkSources_1125893_23.stderr parsl.dataflow.dflow DEBUG: Task 3227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3227 has memoization hash 78bc22eedcb0c8f8a5304689afd7b438 parsl.dataflow.memoization INFO: Task 3227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ed6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3227 try 0 launched on executor multi with executor id 3228 parsl.dataflow.dflow INFO: Standard output for task 3227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e4cf8b8c-9015-45b3-883a-2baeb179cdb3_isrForCrosstalkSources_1125942_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e4cf8b8c-9015-45b3-883a-2baeb179cdb3_isrForCrosstalkSources_1125942_49.stderr parsl.dataflow.dflow DEBUG: Task 3228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3228 has memoization hash c4e5a78b798898e43fc2989babdb6814 parsl.dataflow.memoization INFO: Task 3228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ed800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3228 try 0 launched on executor multi with executor id 3229 parsl.dataflow.dflow INFO: Standard output for task 3228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/5ea115ff-61df-4e77-b650-a343924d47b3_isrForCrosstalkSources_1125913_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/5ea115ff-61df-4e77-b650-a343924d47b3_isrForCrosstalkSources_1125913_49.stderr parsl.dataflow.dflow DEBUG: Task 3229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3229 has memoization hash 7d1cd894906d2493d4801e9ab2f3808b parsl.dataflow.memoization INFO: Task 3229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3229 try 0 launched on executor multi with executor id 3230 parsl.dataflow.dflow INFO: Standard output for task 3229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/d63aa3c0-bc93-4dbf-b47d-293fe39ead56_isrForCrosstalkSources_1125910_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/d63aa3c0-bc93-4dbf-b47d-293fe39ead56_isrForCrosstalkSources_1125910_3.stderr parsl.dataflow.dflow DEBUG: Task 3230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3230 has memoization hash 3e258053d3161118a83a3fd9bc2ca2f9 parsl.dataflow.memoization INFO: Task 3230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3230 try 0 launched on executor multi with executor id 3231 parsl.dataflow.dflow INFO: Standard output for task 3230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/02af419d-75c7-453d-bf1d-b966504fd19d_isrForCrosstalkSources_1125891_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/02af419d-75c7-453d-bf1d-b966504fd19d_isrForCrosstalkSources_1125891_24.stderr parsl.dataflow.dflow DEBUG: Task 3231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3231 has memoization hash fc3ecb9b5069d9ca1e5f58c872b41d4a parsl.dataflow.memoization INFO: Task 3231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3231 try 0 launched on executor multi with executor id 3232 parsl.dataflow.dflow INFO: Standard output for task 3231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/7578d3bb-a0f3-4173-95ae-47bfc3c12fd2_isrForCrosstalkSources_1125889_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/7578d3bb-a0f3-4173-95ae-47bfc3c12fd2_isrForCrosstalkSources_1125889_38.stderr parsl.dataflow.dflow DEBUG: Task 3232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3232 has memoization hash 540ca4207b832d5d1e885f29cee6e079 parsl.dataflow.memoization INFO: Task 3232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3232 try 0 launched on executor multi with executor id 3233 parsl.dataflow.dflow INFO: Standard output for task 3232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/1ee99885-f432-4941-a3ce-1903154d7178_isrForCrosstalkSources_1125911_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/1ee99885-f432-4941-a3ce-1903154d7178_isrForCrosstalkSources_1125911_59.stderr parsl.dataflow.dflow DEBUG: Task 3233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3233 has memoization hash df9936cc32f1ede9aa305a4150a5bb38 parsl.dataflow.memoization INFO: Task 3233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3233 try 0 launched on executor multi with executor id 3234 parsl.dataflow.dflow INFO: Standard output for task 3233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/029f4c92-1cb2-4b53-be97-806c6c6c20f5_isrForCrosstalkSources_1125906_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/029f4c92-1cb2-4b53-be97-806c6c6c20f5_isrForCrosstalkSources_1125906_29.stderr parsl.dataflow.dflow DEBUG: Task 3234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3234 has memoization hash a545642e77ff1313125792b37d696d5b parsl.dataflow.memoization INFO: Task 3234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3234 try 0 launched on executor multi with executor id 3235 parsl.dataflow.dflow INFO: Standard output for task 3234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/808aa5bc-dbca-453a-a23f-c98065d4b1a4_isrForCrosstalkSources_1125938_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/808aa5bc-dbca-453a-a23f-c98065d4b1a4_isrForCrosstalkSources_1125938_1.stderr parsl.dataflow.dflow DEBUG: Task 3235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3235 has memoization hash 3efbafcff59dd0be0cbb8bb028b135a0 parsl.dataflow.memoization INFO: Task 3235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3235 try 0 launched on executor multi with executor id 3236 parsl.dataflow.dflow INFO: Standard output for task 3235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/6e99b67f-1ec7-40c6-8ed9-cadc73e3154b_isrForCrosstalkSources_1125944_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/6e99b67f-1ec7-40c6-8ed9-cadc73e3154b_isrForCrosstalkSources_1125944_51.stderr parsl.dataflow.dflow DEBUG: Task 3236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3236 has memoization hash 620d5ec528dbbf5b630e1f3b11e58294 parsl.dataflow.memoization INFO: Task 3236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3236 try 0 launched on executor multi with executor id 3237 parsl.dataflow.dflow INFO: Standard output for task 3236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/b29922c0-df74-47bf-af3d-c2f09a3add74_isrForCrosstalkSources_1125909_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/b29922c0-df74-47bf-af3d-c2f09a3add74_isrForCrosstalkSources_1125909_3.stderr parsl.dataflow.dflow DEBUG: Task 3237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3237 has memoization hash af318365b4fd4239725cfcaba072f2a3 parsl.dataflow.memoization INFO: Task 3237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3237 try 0 launched on executor multi with executor id 3238 parsl.dataflow.dflow INFO: Standard output for task 3237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/bcfdcca6-ecc6-4ec4-9c32-3d3e4ed8454d_isrForCrosstalkSources_1125902_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/bcfdcca6-ecc6-4ec4-9c32-3d3e4ed8454d_isrForCrosstalkSources_1125902_53.stderr parsl.dataflow.dflow DEBUG: Task 3238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3238 has memoization hash 6bab9aa23b4e0a392fa4f57009a74528 parsl.dataflow.memoization INFO: Task 3238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392868e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3238 try 0 launched on executor multi with executor id 3239 parsl.dataflow.dflow INFO: Standard output for task 3238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/1aabc82b-1c0b-4d3c-b2ab-851c26f271b7_isrForCrosstalkSources_1125913_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/1aabc82b-1c0b-4d3c-b2ab-851c26f271b7_isrForCrosstalkSources_1125913_34.stderr parsl.dataflow.dflow DEBUG: Task 3239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3239 has memoization hash b8158ce449668d93bf6b980e7e5d7b4b parsl.dataflow.memoization INFO: Task 3239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3239 try 0 launched on executor multi with executor id 3240 parsl.dataflow.dflow INFO: Standard output for task 3239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/3263f196-9f39-46ba-a92f-dbf72970a5c3_isrForCrosstalkSources_1125917_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/3263f196-9f39-46ba-a92f-dbf72970a5c3_isrForCrosstalkSources_1125917_16.stderr parsl.dataflow.dflow DEBUG: Task 3240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3240 has memoization hash a1820e5cc10d819191e5207d2c06339f parsl.dataflow.memoization INFO: Task 3240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3240 try 0 launched on executor multi with executor id 3241 parsl.dataflow.dflow INFO: Standard output for task 3240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/b21cc217-a015-4e99-91ed-01c67eb6c6af_isrForCrosstalkSources_1125890_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/b21cc217-a015-4e99-91ed-01c67eb6c6af_isrForCrosstalkSources_1125890_22.stderr parsl.dataflow.dflow DEBUG: Task 3241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3241 has memoization hash ff9d36bbeef647306f29673ea0b9164c parsl.dataflow.memoization INFO: Task 3241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3241 try 0 launched on executor multi with executor id 3242 parsl.dataflow.dflow INFO: Standard output for task 3241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/88168534-f2fd-485e-b28e-68ee9811f893_isrForCrosstalkSources_1125914_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/88168534-f2fd-485e-b28e-68ee9811f893_isrForCrosstalkSources_1125914_24.stderr parsl.dataflow.dflow DEBUG: Task 3242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3242 has memoization hash 8443f383a7b241e882e4087f74c3b819 parsl.dataflow.memoization INFO: Task 3242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392867a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3242 try 0 launched on executor multi with executor id 3243 parsl.dataflow.dflow INFO: Standard output for task 3242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/cca6fdd2-0956-45ad-bdaf-910e66384e0b_isrForCrosstalkSources_1125921_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/cca6fdd2-0956-45ad-bdaf-910e66384e0b_isrForCrosstalkSources_1125921_4.stderr parsl.dataflow.dflow DEBUG: Task 3243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3243 has memoization hash dbaa50a762ed2c4bb018dfb9726ee41f parsl.dataflow.memoization INFO: Task 3243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3243 try 0 launched on executor multi with executor id 3244 parsl.dataflow.dflow INFO: Standard output for task 3243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/58811115-28f0-42f5-8dab-5cd3c8f983ec_isrForCrosstalkSources_1125915_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/58811115-28f0-42f5-8dab-5cd3c8f983ec_isrForCrosstalkSources_1125915_9.stderr parsl.dataflow.dflow DEBUG: Task 3244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3244 has memoization hash 6df630428a6e61fb30a6f0ae90104caf parsl.dataflow.memoization INFO: Task 3244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3244 try 0 launched on executor multi with executor id 3245 parsl.dataflow.dflow INFO: Standard output for task 3244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/9ee490be-4b5b-4a10-b112-6257477e6b01_isrForCrosstalkSources_1125907_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/9ee490be-4b5b-4a10-b112-6257477e6b01_isrForCrosstalkSources_1125907_35.stderr parsl.dataflow.dflow DEBUG: Task 3245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3245 has memoization hash 82d0bafe6d405ae852147d1af997b371 parsl.dataflow.memoization INFO: Task 3245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392865c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3245 try 0 launched on executor multi with executor id 3246 parsl.dataflow.dflow INFO: Standard output for task 3245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/7352328f-493f-4c7f-a4a4-784eea8a5797_isrForCrosstalkSources_1125926_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/7352328f-493f-4c7f-a4a4-784eea8a5797_isrForCrosstalkSources_1125926_18.stderr parsl.dataflow.dflow DEBUG: Task 3246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3246 has memoization hash 01ae455b0875930fd048e91326446068 parsl.dataflow.memoization INFO: Task 3246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3246 try 0 launched on executor multi with executor id 3247 parsl.dataflow.dflow INFO: Standard output for task 3246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e3063182-1282-4a5f-acc0-eb77abef087b_isrForCrosstalkSources_1125895_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e3063182-1282-4a5f-acc0-eb77abef087b_isrForCrosstalkSources_1125895_33.stderr parsl.dataflow.dflow DEBUG: Task 3247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3247 has memoization hash 170bfa68ab2d2a229afbfaa1955e24f4 parsl.dataflow.memoization INFO: Task 3247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3247 try 0 launched on executor multi with executor id 3248 parsl.dataflow.dflow INFO: Standard output for task 3247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/129919c0-00f2-48b1-b85b-0ad8366b161e_isrForCrosstalkSources_1125910_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/129919c0-00f2-48b1-b85b-0ad8366b161e_isrForCrosstalkSources_1125910_6.stderr parsl.dataflow.dflow DEBUG: Task 3248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3248 has memoization hash 074f6fc4d623ed24223bb96c37b12705 parsl.dataflow.memoization INFO: Task 3248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3248 try 0 launched on executor multi with executor id 3249 parsl.dataflow.dflow INFO: Standard output for task 3248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/806e9755-5394-41d7-b7fd-58d15d34ccd3_isrForCrosstalkSources_1125905_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/806e9755-5394-41d7-b7fd-58d15d34ccd3_isrForCrosstalkSources_1125905_2.stderr parsl.dataflow.dflow DEBUG: Task 3249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3249 has memoization hash de89b77c081f9ed6caa451646662f0bc parsl.dataflow.memoization INFO: Task 3249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392863e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3249 try 0 launched on executor multi with executor id 3250 parsl.dataflow.dflow INFO: Standard output for task 3249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b06ef118-70ff-478f-a82f-94c60063df10_isrForCrosstalkSources_1125938_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b06ef118-70ff-478f-a82f-94c60063df10_isrForCrosstalkSources_1125938_48.stderr parsl.dataflow.dflow DEBUG: Task 3250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3250 has memoization hash 93d1775aedf58d739a21ae35518942c9 parsl.dataflow.memoization INFO: Task 3250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3250 try 0 launched on executor multi with executor id 3251 parsl.dataflow.dflow INFO: Standard output for task 3250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/f286d2cf-afa6-4704-aa8b-6c76e4edc359_isrForCrosstalkSources_1125941_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/f286d2cf-afa6-4704-aa8b-6c76e4edc359_isrForCrosstalkSources_1125941_58.stderr parsl.dataflow.dflow DEBUG: Task 3251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3251 has memoization hash 13de7e92082c476880d29ad98d7ccf08 parsl.dataflow.memoization INFO: Task 3251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3251 try 0 launched on executor multi with executor id 3252 parsl.dataflow.dflow INFO: Standard output for task 3251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/a40e3d07-3229-4942-9497-375ad8663182_isrForCrosstalkSources_1125892_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/a40e3d07-3229-4942-9497-375ad8663182_isrForCrosstalkSources_1125892_44.stderr parsl.dataflow.dflow DEBUG: Task 3252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3252 has memoization hash a46553780df80d9a7ddcc1dd9f431b18 parsl.dataflow.memoization INFO: Task 3252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3252 try 0 launched on executor multi with executor id 3253 parsl.dataflow.dflow INFO: Standard output for task 3252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/a6572254-5dda-4af7-b4a1-d81e8b3163d9_isrForCrosstalkSources_1125946_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/a6572254-5dda-4af7-b4a1-d81e8b3163d9_isrForCrosstalkSources_1125946_13.stderr parsl.dataflow.dflow DEBUG: Task 3253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3253 has memoization hash aa953bc81be3f853fec7cd078a753305 parsl.dataflow.memoization INFO: Task 3253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392879c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3253 try 0 launched on executor multi with executor id 3254 parsl.dataflow.dflow INFO: Standard output for task 3253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ae548a9f-a255-418d-86eb-49dd03814f2b_isrForCrosstalkSources_1125918_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ae548a9f-a255-418d-86eb-49dd03814f2b_isrForCrosstalkSources_1125918_23.stderr parsl.dataflow.dflow DEBUG: Task 3254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3254 has memoization hash 53b6e9d112b4b18cc80af07c26509809 parsl.dataflow.memoization INFO: Task 3254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3254 try 0 launched on executor multi with executor id 3255 parsl.dataflow.dflow INFO: Standard output for task 3254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e141419e-9aa5-4f31-aeab-d2614d09a1ff_isrForCrosstalkSources_1125937_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e141419e-9aa5-4f31-aeab-d2614d09a1ff_isrForCrosstalkSources_1125937_28.stderr parsl.dataflow.dflow DEBUG: Task 3255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3255 has memoization hash abd4a14698bd3634233c9d86409f26b1 parsl.dataflow.memoization INFO: Task 3255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3255 try 0 launched on executor multi with executor id 3256 parsl.dataflow.dflow INFO: Standard output for task 3255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/b50bc973-cba0-419f-a51b-cddaff2ace9b_isrForCrosstalkSources_1125946_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/b50bc973-cba0-419f-a51b-cddaff2ace9b_isrForCrosstalkSources_1125946_58.stderr parsl.dataflow.dflow DEBUG: Task 3256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3256 has memoization hash 23d11dd7204c009efe367f8b85c7559f parsl.dataflow.memoization INFO: Task 3256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3256 try 0 launched on executor multi with executor id 3257 parsl.dataflow.dflow INFO: Standard output for task 3256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/ba5686f3-a0c3-4e9f-9935-82e630ce069a_isrForCrosstalkSources_1125893_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/ba5686f3-a0c3-4e9f-9935-82e630ce069a_isrForCrosstalkSources_1125893_42.stderr parsl.dataflow.dflow DEBUG: Task 3257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3257 has memoization hash 035a6278b07fc422fb5a43f478479345 parsl.dataflow.memoization INFO: Task 3257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392877e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3257 try 0 launched on executor multi with executor id 3258 parsl.dataflow.dflow INFO: Standard output for task 3257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/13790a49-9050-4c2c-93e8-9640f21cda5c_isrForCrosstalkSources_1125908_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/13790a49-9050-4c2c-93e8-9640f21cda5c_isrForCrosstalkSources_1125908_11.stderr parsl.dataflow.dflow DEBUG: Task 3258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3258 has memoization hash 0340b7ce8edd4e45d43e1454040b25b7 parsl.dataflow.memoization INFO: Task 3258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392876a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3258 try 0 launched on executor multi with executor id 3259 parsl.dataflow.dflow INFO: Standard output for task 3258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/61eae751-0125-4098-8e7b-e657177c76ec_isrForCrosstalkSources_1125895_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/61eae751-0125-4098-8e7b-e657177c76ec_isrForCrosstalkSources_1125895_34.stderr parsl.dataflow.dflow DEBUG: Task 3259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3259 has memoization hash c79d95e92fc1d244c61dda41a84f086c parsl.dataflow.memoization INFO: Task 3259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3259 try 0 launched on executor multi with executor id 3260 parsl.dataflow.dflow INFO: Standard output for task 3259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c1f31d53-9273-43cb-82d7-42083aa61d3d_isrForCrosstalkSources_1125893_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c1f31d53-9273-43cb-82d7-42083aa61d3d_isrForCrosstalkSources_1125893_17.stderr parsl.dataflow.dflow DEBUG: Task 3260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3260 has memoization hash f61345224af95fb104c96d5b0959ffe4 parsl.dataflow.memoization INFO: Task 3260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3260 try 0 launched on executor multi with executor id 3261 parsl.dataflow.dflow INFO: Standard output for task 3260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/0e86816c-37cd-4284-afe0-c6b674d3c7a1_isrForCrosstalkSources_1125888_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/0e86816c-37cd-4284-afe0-c6b674d3c7a1_isrForCrosstalkSources_1125888_13.stderr parsl.dataflow.dflow DEBUG: Task 3261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3261 has memoization hash 6a8934ad51b5523ee09b22b165d39c70 parsl.dataflow.memoization INFO: Task 3261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:06 proc_lsst.multi:170 [INFO] [multi] job_id 16896440 proc_lsst.multi INFO: [multi] job_id 16896440 2024-03-07 15:07:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 15:07:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:06 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896440 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896440 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 16896440 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-07 15:07:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-07 15:07:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 15:07:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3261 try 0 launched on executor multi with executor id 3262 parsl.dataflow.dflow INFO: Standard output for task 3261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/8acbdf7a-ea03-4c3b-9316-8b52cfe639cc_isrForCrosstalkSources_1125926_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/8acbdf7a-ea03-4c3b-9316-8b52cfe639cc_isrForCrosstalkSources_1125926_52.stderr parsl.dataflow.dflow DEBUG: Task 3262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3262 has memoization hash 2dd3ea19649db5c3a20fcba8c6b572f9 parsl.dataflow.memoization INFO: Task 3262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392874c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3262 try 0 launched on executor multi with executor id 3263 parsl.dataflow.dflow INFO: Standard output for task 3262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/aaf8a76f-6608-4339-8b1c-a281aedc61d4_isrForCrosstalkSources_1125941_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/aaf8a76f-6608-4339-8b1c-a281aedc61d4_isrForCrosstalkSources_1125941_27.stderr parsl.dataflow.dflow DEBUG: Task 3263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3263 has memoization hash 38f35800b9c13dadfb29bb861f2b74dc parsl.dataflow.memoization INFO: Task 3263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39285ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3263 try 0 launched on executor multi with executor id 3264 parsl.dataflow.dflow INFO: Standard output for task 3263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/f54a8f16-3654-4af7-9ec5-61f25d27a3d1_isrForCrosstalkSources_1125937_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/f54a8f16-3654-4af7-9ec5-61f25d27a3d1_isrForCrosstalkSources_1125937_11.stderr parsl.dataflow.dflow DEBUG: Task 3264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3264 has memoization hash 965c98c9fe206771dbab0619599e496e parsl.dataflow.memoization INFO: Task 3264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3264 try 0 launched on executor multi with executor id 3265 parsl.dataflow.dflow INFO: Standard output for task 3264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/fb75d4ab-2478-4443-90d7-67a983917978_isrForCrosstalkSources_1125912_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/fb75d4ab-2478-4443-90d7-67a983917978_isrForCrosstalkSources_1125912_21.stderr parsl.dataflow.dflow DEBUG: Task 3265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3265 has memoization hash 9d81e1e9bc56384cd551903ddd54bc8e parsl.dataflow.memoization INFO: Task 3265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3265 try 0 launched on executor multi with executor id 3266 parsl.dataflow.dflow INFO: Standard output for task 3265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/70279cff-afb4-4cc4-a07f-1c0bd69fd594_isrForCrosstalkSources_1125913_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/70279cff-afb4-4cc4-a07f-1c0bd69fd594_isrForCrosstalkSources_1125913_33.stderr parsl.dataflow.dflow DEBUG: Task 3266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3266 has memoization hash 4727408ef66b213b01634b6273eefd2e parsl.dataflow.memoization INFO: Task 3266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39287060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3266 try 0 launched on executor multi with executor id 3267 parsl.dataflow.dflow INFO: Standard output for task 3266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/9c092ac2-6d24-482a-844b-124c96c93a61_isrForCrosstalkSources_1125937_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/9c092ac2-6d24-482a-844b-124c96c93a61_isrForCrosstalkSources_1125937_1.stderr parsl.dataflow.dflow DEBUG: Task 3267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3267 has memoization hash d7b7527902dd56e67ee0cb4192a16004 parsl.dataflow.memoization INFO: Task 3267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3267 try 0 launched on executor multi with executor id 3268 parsl.dataflow.dflow INFO: Standard output for task 3267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/8b871836-658b-4bfc-807e-f9d10e685bd7_isrForCrosstalkSources_1125928_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/8b871836-658b-4bfc-807e-f9d10e685bd7_isrForCrosstalkSources_1125928_23.stderr parsl.dataflow.dflow DEBUG: Task 3268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3268 has memoization hash 4546548114b13a650d1f64cc30b25464 parsl.dataflow.memoization INFO: Task 3268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3268 try 0 launched on executor multi with executor id 3269 parsl.dataflow.dflow INFO: Standard output for task 3268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/8dc859af-5e59-427c-adc2-6b0d8f4207d5_isrForCrosstalkSources_1125904_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/8dc859af-5e59-427c-adc2-6b0d8f4207d5_isrForCrosstalkSources_1125904_11.stderr parsl.dataflow.dflow DEBUG: Task 3269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3269 has memoization hash 08b82fe3c7752719b9d00c22e42c66b1 parsl.dataflow.memoization INFO: Task 3269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3269 try 0 launched on executor multi with executor id 3270 parsl.dataflow.dflow INFO: Standard output for task 3269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/e55d8f3b-d464-4fb5-99fe-82d941a5ddf8_isrForCrosstalkSources_1125893_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/e55d8f3b-d464-4fb5-99fe-82d941a5ddf8_isrForCrosstalkSources_1125893_52.stderr parsl.dataflow.dflow DEBUG: Task 3270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3270 has memoization hash f0ceca61ff0feea1b97d8d55557bfc5a parsl.dataflow.memoization INFO: Task 3270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392860c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3270 try 0 launched on executor multi with executor id 3271 parsl.dataflow.dflow INFO: Standard output for task 3270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4aa09ddf-7153-4dec-a861-958eb78e6c69_isrForCrosstalkSources_1125908_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4aa09ddf-7153-4dec-a861-958eb78e6c69_isrForCrosstalkSources_1125908_8.stderr parsl.dataflow.dflow DEBUG: Task 3271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3271 has memoization hash 5d9482f79352bb7ee213c56d64db80b8 parsl.dataflow.memoization INFO: Task 3271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3271 try 0 launched on executor multi with executor id 3272 parsl.dataflow.dflow INFO: Standard output for task 3271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/a790d3cd-6aa3-4f51-8e73-ff8344a6212d_isrForCrosstalkSources_1125890_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/a790d3cd-6aa3-4f51-8e73-ff8344a6212d_isrForCrosstalkSources_1125890_44.stderr parsl.dataflow.dflow DEBUG: Task 3272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3272 has memoization hash 9557b91e117fa8bbac641da7e7e2d8c6 parsl.dataflow.memoization INFO: Task 3272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39286480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3272 try 0 launched on executor multi with executor id 3273 parsl.dataflow.dflow INFO: Standard output for task 3272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/d54975bb-27b7-4fab-b187-4463c041de54_isrForCrosstalkSources_1125920_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/d54975bb-27b7-4fab-b187-4463c041de54_isrForCrosstalkSources_1125920_25.stderr parsl.dataflow.dflow DEBUG: Task 3273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3273 has memoization hash a4ec987c43a3ec160820e5d4449faa01 parsl.dataflow.memoization INFO: Task 3273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392862a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3273 try 0 launched on executor multi with executor id 3274 parsl.dataflow.dflow INFO: Standard output for task 3273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/04ec1663-aa41-47c5-ae65-ced70af7fdba_isrForCrosstalkSources_1125932_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/04ec1663-aa41-47c5-ae65-ced70af7fdba_isrForCrosstalkSources_1125932_44.stderr parsl.dataflow.dflow DEBUG: Task 3274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3274 has memoization hash 4ca3bf7618b9c7775b1563b049e331d5 parsl.dataflow.memoization INFO: Task 3274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3274 try 0 launched on executor multi with executor id 3275 parsl.dataflow.dflow INFO: Standard output for task 3274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2584c6cb-2ad7-4825-b9a8-ca12ef573ab1_isrForCrosstalkSources_1125933_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2584c6cb-2ad7-4825-b9a8-ca12ef573ab1_isrForCrosstalkSources_1125933_44.stderr parsl.dataflow.dflow DEBUG: Task 3275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3275 has memoization hash 219d1b93f750bbd0a32e38eb964145f6 parsl.dataflow.memoization INFO: Task 3275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ef880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3275 try 0 launched on executor multi with executor id 3276 parsl.dataflow.dflow INFO: Standard output for task 3275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/c30679c6-5e32-48ce-ba3a-ef88467b5547_isrForCrosstalkSources_1125934_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/c30679c6-5e32-48ce-ba3a-ef88467b5547_isrForCrosstalkSources_1125934_10.stderr parsl.dataflow.dflow DEBUG: Task 3276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3276 has memoization hash 6aa2528dd8559e1cc3fa092a5d346c51 parsl.dataflow.memoization INFO: Task 3276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ec180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3276 try 0 launched on executor multi with executor id 3277 parsl.dataflow.dflow INFO: Standard output for task 3276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3a3606e8-1bef-4491-b19c-3fd021419755_isrForCrosstalkSources_1125920_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3a3606e8-1bef-4491-b19c-3fd021419755_isrForCrosstalkSources_1125920_47.stderr parsl.dataflow.dflow DEBUG: Task 3277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3277 has memoization hash e5d48c0adadf381c52f7755624b40bce parsl.dataflow.memoization INFO: Task 3277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ec040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3277 try 0 launched on executor multi with executor id 3278 parsl.dataflow.dflow INFO: Standard output for task 3277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/fba49205-95a7-41ac-bd26-11b9d9326e54_isrForCrosstalkSources_1125945_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/fba49205-95a7-41ac-bd26-11b9d9326e54_isrForCrosstalkSources_1125945_55.stderr parsl.dataflow.dflow DEBUG: Task 3278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3278 has memoization hash 32d2ccd70abe651b19a521e9e81b8110 parsl.dataflow.memoization INFO: Task 3278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ec4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3278 try 0 launched on executor multi with executor id 3279 parsl.dataflow.dflow INFO: Standard output for task 3278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/96a2065b-6818-45a6-a096-2f2fb29ece9f_isrForCrosstalkSources_1125940_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/96a2065b-6818-45a6-a096-2f2fb29ece9f_isrForCrosstalkSources_1125940_13.stderr parsl.dataflow.dflow DEBUG: Task 3279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3279 has memoization hash 2c0d989f3b0e8eb2e53a78ded61d60d5 parsl.dataflow.memoization INFO: Task 3279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ef920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3279 try 0 launched on executor multi with executor id 3280 parsl.dataflow.dflow INFO: Standard output for task 3279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/91c73496-b541-46e0-82b6-5d3e998764d3_isrForCrosstalkSources_1125896_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/91c73496-b541-46e0-82b6-5d3e998764d3_isrForCrosstalkSources_1125896_51.stderr parsl.dataflow.dflow DEBUG: Task 3280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3280 has memoization hash 8dcc5ad4f0ead103532f8ad200ffb996 parsl.dataflow.memoization INFO: Task 3280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ed4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3280 try 0 launched on executor multi with executor id 3281 parsl.dataflow.dflow INFO: Standard output for task 3280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/6a280309-5d57-42b8-9458-0420c78fa624_isrForCrosstalkSources_1125905_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/6a280309-5d57-42b8-9458-0420c78fa624_isrForCrosstalkSources_1125905_31.stderr parsl.dataflow.dflow DEBUG: Task 3281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3281 has memoization hash 4c34aebbafaf883f3c7a79d38e22bf8d parsl.dataflow.memoization INFO: Task 3281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ed440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3281 try 0 launched on executor multi with executor id 3282 parsl.dataflow.dflow INFO: Standard output for task 3281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/7f257e46-86af-48a3-bcb1-30ab6a61d6c7_isrForCrosstalkSources_1125917_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/7f257e46-86af-48a3-bcb1-30ab6a61d6c7_isrForCrosstalkSources_1125917_8.stderr parsl.dataflow.dflow DEBUG: Task 3282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3282 has memoization hash 624772d35e35d9422ec23845da5b3530 parsl.dataflow.memoization INFO: Task 3282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ec360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3282 try 0 launched on executor multi with executor id 3283 parsl.dataflow.dflow INFO: Standard output for task 3282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/fbd3f7b4-2ec0-4735-8d94-db8258f2e21f_isrForCrosstalkSources_1125937_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/fbd3f7b4-2ec0-4735-8d94-db8258f2e21f_isrForCrosstalkSources_1125937_33.stderr parsl.dataflow.dflow DEBUG: Task 3283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3283 has memoization hash 54d39a33ec081cb08d342c3fd77584bd parsl.dataflow.memoization INFO: Task 3283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ed300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3283 try 0 launched on executor multi with executor id 3284 parsl.dataflow.dflow INFO: Standard output for task 3283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3ec6298a-bf5f-4560-80b3-07e321e2f0f8_isrForCrosstalkSources_1125941_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3ec6298a-bf5f-4560-80b3-07e321e2f0f8_isrForCrosstalkSources_1125941_6.stderr parsl.dataflow.dflow DEBUG: Task 3284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3284 has memoization hash 761bb33d6dae6d71322240c054b663e2 parsl.dataflow.memoization INFO: Task 3284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ed3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3284 try 0 launched on executor multi with executor id 3285 parsl.dataflow.dflow INFO: Standard output for task 3284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/399350b1-e543-45e4-834b-68ee9d710cd9_isrForCrosstalkSources_1125924_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/399350b1-e543-45e4-834b-68ee9d710cd9_isrForCrosstalkSources_1125924_59.stderr parsl.dataflow.dflow DEBUG: Task 3285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3285 has memoization hash 8347e15c0361f80c13415accf20d02c8 parsl.dataflow.memoization INFO: Task 3285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ed260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3285 try 0 launched on executor multi with executor id 3286 parsl.dataflow.dflow INFO: Standard output for task 3285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/614989e7-d633-4686-99ca-299330c827c0_isrForCrosstalkSources_1125905_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/614989e7-d633-4686-99ca-299330c827c0_isrForCrosstalkSources_1125905_10.stderr parsl.dataflow.dflow DEBUG: Task 3286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3286 has memoization hash 0701ce9100d67641ace1a28449d07c4a parsl.dataflow.memoization INFO: Task 3286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ed120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3286 try 0 launched on executor multi with executor id 3287 parsl.dataflow.dflow INFO: Standard output for task 3286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/37ab7dc9-e7ec-41da-a85e-3ab00d658d2a_isrForCrosstalkSources_1125929_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/37ab7dc9-e7ec-41da-a85e-3ab00d658d2a_isrForCrosstalkSources_1125929_53.stderr parsl.dataflow.dflow DEBUG: Task 3287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3287 has memoization hash 8d6dfc370b5c34a10a47b0e070a6c7c3 parsl.dataflow.memoization INFO: Task 3287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ed080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3287 try 0 launched on executor multi with executor id 3288 parsl.dataflow.dflow INFO: Standard output for task 3287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/9be8919e-f10b-458d-8f21-0702d78c6f80_isrForCrosstalkSources_1125905_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/9be8919e-f10b-458d-8f21-0702d78c6f80_isrForCrosstalkSources_1125905_14.stderr parsl.dataflow.dflow DEBUG: Task 3288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3288 has memoization hash c559e3ed5db63affbeb415ce0435f235 parsl.dataflow.memoization INFO: Task 3288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ecfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3288 try 0 launched on executor multi with executor id 3289 parsl.dataflow.dflow INFO: Standard output for task 3288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/74778877-e0ec-4a64-a402-bf612f18e8d2_isrForCrosstalkSources_1125909_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/74778877-e0ec-4a64-a402-bf612f18e8d2_isrForCrosstalkSources_1125909_34.stderr parsl.dataflow.dflow DEBUG: Task 3289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3289 has memoization hash 5693db3db404822b7d615dbf2e64678a parsl.dataflow.memoization INFO: Task 3289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ed1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3289 try 0 launched on executor multi with executor id 3290 parsl.dataflow.dflow INFO: Standard output for task 3289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/dfd2e7fc-1c9f-42a3-8713-b993a633e6c0_isrForCrosstalkSources_1125910_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/dfd2e7fc-1c9f-42a3-8713-b993a633e6c0_isrForCrosstalkSources_1125910_45.stderr parsl.dataflow.dflow DEBUG: Task 3290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3290 has memoization hash a3fec94400acc14f0600ad23b566ed2c parsl.dataflow.memoization INFO: Task 3290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ecf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3290 try 0 launched on executor multi with executor id 3291 parsl.dataflow.dflow INFO: Standard output for task 3290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/bbfd1090-92aa-4f08-a037-07c1a18b43a6_isrForCrosstalkSources_1125910_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/bbfd1090-92aa-4f08-a037-07c1a18b43a6_isrForCrosstalkSources_1125910_14.stderr parsl.dataflow.dflow DEBUG: Task 3291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3291 has memoization hash 1799c9426ae156fccd0fd7ba18edfef5 parsl.dataflow.memoization INFO: Task 3291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ecea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3291 try 0 launched on executor multi with executor id 3292 parsl.dataflow.dflow INFO: Standard output for task 3291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/1e15539c-cfa4-45c7-bf68-063fe0af6dd7_isrForCrosstalkSources_1125892_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/1e15539c-cfa4-45c7-bf68-063fe0af6dd7_isrForCrosstalkSources_1125892_59.stderr parsl.dataflow.dflow DEBUG: Task 3292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3292 has memoization hash 2b074f7f34728330a78f13f94936b8e0 parsl.dataflow.memoization INFO: Task 3292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ecd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3292 try 0 launched on executor multi with executor id 3293 parsl.dataflow.dflow INFO: Standard output for task 3292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/a0fd27ff-86b0-49c6-82b3-d37c6c0f0dc4_isrForCrosstalkSources_1125908_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/a0fd27ff-86b0-49c6-82b3-d37c6c0f0dc4_isrForCrosstalkSources_1125908_43.stderr parsl.dataflow.dflow DEBUG: Task 3293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3293 has memoization hash ab48273b986c69d080e0202ddbff8b49 parsl.dataflow.memoization INFO: Task 3293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ecc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3293 try 0 launched on executor multi with executor id 3294 parsl.dataflow.dflow INFO: Standard output for task 3293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/e34ab16e-a183-4627-ab5f-c952d4ea0aab_isrForCrosstalkSources_1125938_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/e34ab16e-a183-4627-ab5f-c952d4ea0aab_isrForCrosstalkSources_1125938_2.stderr parsl.dataflow.dflow DEBUG: Task 3294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3294 has memoization hash e34838800794f2061c6ca5b4fa03b1d5 parsl.dataflow.memoization INFO: Task 3294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392eccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3294 try 0 launched on executor multi with executor id 3295 parsl.dataflow.dflow INFO: Standard output for task 3294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/178c4800-8039-4107-aa47-4d3988cc13df_isrForCrosstalkSources_1125921_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/178c4800-8039-4107-aa47-4d3988cc13df_isrForCrosstalkSources_1125921_60.stderr parsl.dataflow.dflow DEBUG: Task 3295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3295 has memoization hash 80aa53196123728a7710a6e1dd172e91 parsl.dataflow.memoization INFO: Task 3295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ece00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3295 try 0 launched on executor multi with executor id 3296 parsl.dataflow.dflow INFO: Standard output for task 3295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/58e112fe-7370-4e33-82f0-a9681bc78a72_isrForCrosstalkSources_1125939_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/58e112fe-7370-4e33-82f0-a9681bc78a72_isrForCrosstalkSources_1125939_22.stderr parsl.dataflow.dflow DEBUG: Task 3296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3296 has memoization hash d349ca7e39afb4f65e6e8c186a5ab9fa parsl.dataflow.memoization INFO: Task 3296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ecb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3296 try 0 launched on executor multi with executor id 3297 parsl.dataflow.dflow INFO: Standard output for task 3296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/51758191-cedc-4ab3-aa76-42bd71e1067a_isrForCrosstalkSources_1125921_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/51758191-cedc-4ab3-aa76-42bd71e1067a_isrForCrosstalkSources_1125921_57.stderr parsl.dataflow.dflow DEBUG: Task 3297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3297 has memoization hash 6a828d8714dd242ad6da8ef65ec9731c parsl.dataflow.memoization INFO: Task 3297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392eca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3297 try 0 launched on executor multi with executor id 3298 parsl.dataflow.dflow INFO: Standard output for task 3297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/cd04e677-0bd8-46b9-821f-c53078e1a62d_isrForCrosstalkSources_1125909_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/cd04e677-0bd8-46b9-821f-c53078e1a62d_isrForCrosstalkSources_1125909_31.stderr parsl.dataflow.dflow DEBUG: Task 3298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3298 has memoization hash 5ec30841897d8beb4c3d8faf0433de35 parsl.dataflow.memoization INFO: Task 3298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ec900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3298 try 0 launched on executor multi with executor id 3299 parsl.dataflow.dflow INFO: Standard output for task 3298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/61c48912-562d-48a0-bb61-866aac27570e_isrForCrosstalkSources_1125928_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/61c48912-562d-48a0-bb61-866aac27570e_isrForCrosstalkSources_1125928_62.stderr parsl.dataflow.dflow DEBUG: Task 3299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3299 has memoization hash 5d07d92aac19593bdb2b2597fb35d0be parsl.dataflow.memoization INFO: Task 3299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ec9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3299 try 0 launched on executor multi with executor id 3300 parsl.dataflow.dflow INFO: Standard output for task 3299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/b8a09bbe-5a7d-4e1b-9e15-041a2396cb25_isrForCrosstalkSources_1125891_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/b8a09bbe-5a7d-4e1b-9e15-041a2396cb25_isrForCrosstalkSources_1125891_40.stderr parsl.dataflow.dflow DEBUG: Task 3300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3300 has memoization hash 5dd3e4649ebfb088842001fbbbc74247 parsl.dataflow.memoization INFO: Task 3300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ecae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3300 try 0 launched on executor multi with executor id 3301 parsl.dataflow.dflow INFO: Standard output for task 3300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/7f540807-fc30-4e49-901c-b034977be9c6_isrForCrosstalkSources_1125931_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/7f540807-fc30-4e49-901c-b034977be9c6_isrForCrosstalkSources_1125931_26.stderr parsl.dataflow.dflow DEBUG: Task 3301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3301 has memoization hash 7e6d79aed075200a801d1f8a93dee315 parsl.dataflow.memoization INFO: Task 3301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ec7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3301 try 0 launched on executor multi with executor id 3302 parsl.dataflow.dflow INFO: Standard output for task 3301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/5c16359e-bfac-452c-b2e9-e1cb79ac92b7_isrForCrosstalkSources_1125916_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/5c16359e-bfac-452c-b2e9-e1cb79ac92b7_isrForCrosstalkSources_1125916_53.stderr parsl.dataflow.dflow DEBUG: Task 3302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3302 has memoization hash 554b24dd6b2a8ed8e2452cd52e9c67b4 parsl.dataflow.memoization INFO: Task 3302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ec860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3302 try 0 launched on executor multi with executor id 3303 parsl.dataflow.dflow INFO: Standard output for task 3302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/c3b98ec2-2695-4840-b1c5-2edc477919fe_isrForCrosstalkSources_1125912_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/c3b98ec2-2695-4840-b1c5-2edc477919fe_isrForCrosstalkSources_1125912_27.stderr parsl.dataflow.dflow DEBUG: Task 3303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3303 has memoization hash 28032acc2af8fd2b1ef0dd95f88c4af0 parsl.dataflow.memoization INFO: Task 3303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ec5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3303 try 0 launched on executor multi with executor id 3304 parsl.dataflow.dflow INFO: Standard output for task 3303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/a8e601ad-5235-4acf-aafa-ba9e7e4e858e_isrForCrosstalkSources_1125889_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/a8e601ad-5235-4acf-aafa-ba9e7e4e858e_isrForCrosstalkSources_1125889_23.stderr parsl.dataflow.dflow DEBUG: Task 3304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3304 has memoization hash 8f22981940555b80f445627b0fa42101 parsl.dataflow.memoization INFO: Task 3304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ec720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3304 try 0 launched on executor multi with executor id 3305 parsl.dataflow.dflow INFO: Standard output for task 3304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/2078847b-11b5-4b2b-92b0-4694e4e2a2d2_isrForCrosstalkSources_1125931_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/2078847b-11b5-4b2b-92b0-4694e4e2a2d2_isrForCrosstalkSources_1125931_9.stderr parsl.dataflow.dflow DEBUG: Task 3305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3305 has memoization hash 1d8968eaf1d4759cb12368856fb94e22 parsl.dataflow.memoization INFO: Task 3305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ec680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3305 try 0 launched on executor multi with executor id 3306 parsl.dataflow.dflow INFO: Standard output for task 3305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/41a90ab0-6aba-48f0-a153-1b48a6f594a2_isrForCrosstalkSources_1125906_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/41a90ab0-6aba-48f0-a153-1b48a6f594a2_isrForCrosstalkSources_1125906_35.stderr parsl.dataflow.dflow DEBUG: Task 3306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3306 has memoization hash 84e5215fc472922d3f5caba25a6623e6 parsl.dataflow.memoization INFO: Task 3306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ec400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3306 try 0 launched on executor multi with executor id 3307 parsl.dataflow.dflow INFO: Standard output for task 3306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/c25481ac-a785-455f-a4a5-d4981b9f710b_isrForCrosstalkSources_1125940_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/c25481ac-a785-455f-a4a5-d4981b9f710b_isrForCrosstalkSources_1125940_38.stderr parsl.dataflow.dflow DEBUG: Task 3307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3307 has memoization hash 7d68ca9884d733f3b45441ca3d60488a parsl.dataflow.memoization INFO: Task 3307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ec540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3307 try 0 launched on executor multi with executor id 3308 parsl.dataflow.dflow INFO: Standard output for task 3307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/d3aef5d9-bf0b-46e1-8e6a-37fea01fa4f7_isrForCrosstalkSources_1125920_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/d3aef5d9-bf0b-46e1-8e6a-37fea01fa4f7_isrForCrosstalkSources_1125920_61.stderr parsl.dataflow.dflow DEBUG: Task 3308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3308 has memoization hash d31d14ed7205724c48d72d240d003c64 parsl.dataflow.memoization INFO: Task 3308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392eda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3308 try 0 launched on executor multi with executor id 3309 parsl.dataflow.dflow INFO: Standard output for task 3308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/82265d5f-d1b1-4004-b997-0fd41f59fbf6_isrForCrosstalkSources_1125888_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/82265d5f-d1b1-4004-b997-0fd41f59fbf6_isrForCrosstalkSources_1125888_34.stderr parsl.dataflow.dflow DEBUG: Task 3309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3309 has memoization hash 109342087f4131cc5c362d54bcfe5263 parsl.dataflow.memoization INFO: Task 3309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ed940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3309 try 0 launched on executor multi with executor id 3310 parsl.dataflow.dflow INFO: Standard output for task 3309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/7426e447-c3bc-4c53-82fd-b1f7f2d3b984_isrForCrosstalkSources_1125899_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/7426e447-c3bc-4c53-82fd-b1f7f2d3b984_isrForCrosstalkSources_1125899_15.stderr parsl.dataflow.dflow DEBUG: Task 3310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3310 has memoization hash 351306e463f36585a8925d64876023e4 parsl.dataflow.memoization INFO: Task 3310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ed9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3310 try 0 launched on executor multi with executor id 3311 parsl.dataflow.dflow INFO: Standard output for task 3310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a6345087-82dd-4533-a7ab-4274d1c4927c_isrForCrosstalkSources_1125933_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a6345087-82dd-4533-a7ab-4274d1c4927c_isrForCrosstalkSources_1125933_59.stderr parsl.dataflow.dflow DEBUG: Task 3311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3311 has memoization hash 7a66d28d270bd285d3b21611df8c943d parsl.dataflow.memoization INFO: Task 3311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392edbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3311 try 0 launched on executor multi with executor id 3312 parsl.dataflow.dflow INFO: Standard output for task 3311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/51cef015-83d2-4237-9b8b-e749d94b8a71_isrForCrosstalkSources_1125919_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/51cef015-83d2-4237-9b8b-e749d94b8a71_isrForCrosstalkSources_1125919_51.stderr parsl.dataflow.dflow DEBUG: Task 3312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3312 has memoization hash 2db270f436b8e63da905d86653a30719 parsl.dataflow.memoization INFO: Task 3312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392edb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3312 try 0 launched on executor multi with executor id 3313 parsl.dataflow.dflow INFO: Standard output for task 3312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6f87bc43-bc5d-470f-a650-ad476b0e63ac_isrForCrosstalkSources_1125927_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6f87bc43-bc5d-470f-a650-ad476b0e63ac_isrForCrosstalkSources_1125927_50.stderr parsl.dataflow.dflow DEBUG: Task 3313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3313 has memoization hash 157a1d1c2055651094ee3f9396d30c59 parsl.dataflow.memoization INFO: Task 3313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392edda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3313 try 0 launched on executor multi with executor id 3314 parsl.dataflow.dflow INFO: Standard output for task 3313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d2d81f77-1097-44eb-ae57-6499601d74b3_isrForCrosstalkSources_1125919_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d2d81f77-1097-44eb-ae57-6499601d74b3_isrForCrosstalkSources_1125919_19.stderr parsl.dataflow.dflow DEBUG: Task 3314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3314 has memoization hash f26d3f3aae25aee6dbab05b58de4cb8b parsl.dataflow.memoization INFO: Task 3314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392edd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3314 try 0 launched on executor multi with executor id 3315 parsl.dataflow.dflow INFO: Standard output for task 3314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/f443a57a-8cc1-4802-810f-1f94dbcddbf6_isrForCrosstalkSources_1125911_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/f443a57a-8cc1-4802-810f-1f94dbcddbf6_isrForCrosstalkSources_1125911_50.stderr parsl.dataflow.dflow DEBUG: Task 3315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3315 has memoization hash 33fd8cbb26005ac0aa5c238fd49f534d parsl.dataflow.memoization INFO: Task 3315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392edc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3315 try 0 launched on executor multi with executor id 3316 parsl.dataflow.dflow INFO: Standard output for task 3315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/ace305ff-3534-4aa5-b237-f547560b66f0_isrForCrosstalkSources_1125927_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/ace305ff-3534-4aa5-b237-f547560b66f0_isrForCrosstalkSources_1125927_54.stderr parsl.dataflow.dflow DEBUG: Task 3316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3316 has memoization hash 0cd3f172455b33bdaa7aaadace25b349 parsl.dataflow.memoization INFO: Task 3316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392edee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3316 try 0 launched on executor multi with executor id 3317 parsl.dataflow.dflow INFO: Standard output for task 3316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/48a46a6d-41d9-4f02-bd56-f2005a488423_isrForCrosstalkSources_1125889_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/48a46a6d-41d9-4f02-bd56-f2005a488423_isrForCrosstalkSources_1125889_62.stderr parsl.dataflow.dflow DEBUG: Task 3317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3317 has memoization hash 88c14e6ff354a786bb58474ab51ea957 parsl.dataflow.memoization INFO: Task 3317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392edf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3317 try 0 launched on executor multi with executor id 3318 parsl.dataflow.dflow INFO: Standard output for task 3317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/c5f22cd5-c88a-4c6e-9ce2-29cf427658a8_isrForCrosstalkSources_1125945_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/c5f22cd5-c88a-4c6e-9ce2-29cf427658a8_isrForCrosstalkSources_1125945_32.stderr parsl.dataflow.dflow DEBUG: Task 3318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3318 has memoization hash 322677e58598b0cdb271c7b2151d9bd1 parsl.dataflow.memoization INFO: Task 3318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ede40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3318 try 0 launched on executor multi with executor id 3319 parsl.dataflow.dflow INFO: Standard output for task 3318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/5a810ae5-516f-4cc1-bc4a-2f2d6a7e162a_isrForCrosstalkSources_1125888_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/5a810ae5-516f-4cc1-bc4a-2f2d6a7e162a_isrForCrosstalkSources_1125888_1.stderr parsl.dataflow.dflow DEBUG: Task 3319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3319 has memoization hash a6054da85794b52d898c2743f75e197d parsl.dataflow.memoization INFO: Task 3319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ee020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3319 try 0 launched on executor multi with executor id 3320 parsl.dataflow.dflow INFO: Standard output for task 3319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e8fc3d35-8ece-4f41-b013-524a06106209_isrForCrosstalkSources_1125916_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e8fc3d35-8ece-4f41-b013-524a06106209_isrForCrosstalkSources_1125916_33.stderr parsl.dataflow.dflow DEBUG: Task 3320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3320 has memoization hash c906df3018c67876f2ff5dad2d89f366 parsl.dataflow.memoization INFO: Task 3320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ee160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3320 try 0 launched on executor multi with executor id 3321 parsl.dataflow.dflow INFO: Standard output for task 3320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/d3e4f0e9-dcfa-4b60-b45d-6d35139ce119_isrForCrosstalkSources_1125899_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/d3e4f0e9-dcfa-4b60-b45d-6d35139ce119_isrForCrosstalkSources_1125899_43.stderr parsl.dataflow.dflow DEBUG: Task 3321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3321 has memoization hash 6cef15e9bba015a69c20553439ee01e6 parsl.dataflow.memoization INFO: Task 3321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ee200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3321 try 0 launched on executor multi with executor id 3322 parsl.dataflow.dflow INFO: Standard output for task 3321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/2120f660-6624-4cbf-a0c1-bf9bf2577650_isrForCrosstalkSources_1125930_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/2120f660-6624-4cbf-a0c1-bf9bf2577650_isrForCrosstalkSources_1125930_24.stderr parsl.dataflow.dflow DEBUG: Task 3322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3322 has memoization hash 40a435902b2f786dda23c5b963235765 parsl.dataflow.memoization INFO: Task 3322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ee2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3322 try 0 launched on executor multi with executor id 3323 parsl.dataflow.dflow INFO: Standard output for task 3322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/aee262ce-68d8-43f3-a6be-460100dc7c8d_isrForCrosstalkSources_1125928_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/aee262ce-68d8-43f3-a6be-460100dc7c8d_isrForCrosstalkSources_1125928_6.stderr parsl.dataflow.dflow DEBUG: Task 3323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3323 has memoization hash d81619114084925600ba6622d3c8ee82 parsl.dataflow.memoization INFO: Task 3323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ee340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3323 try 0 launched on executor multi with executor id 3324 parsl.dataflow.dflow INFO: Standard output for task 3323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/32785117-39b5-4a9c-aaf3-4029da2e076b_isrForCrosstalkSources_1125921_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/32785117-39b5-4a9c-aaf3-4029da2e076b_isrForCrosstalkSources_1125921_33.stderr parsl.dataflow.dflow DEBUG: Task 3324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3324 has memoization hash e03925cafe7d5875617f2eb1842b1578 parsl.dataflow.memoization INFO: Task 3324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ee0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:06 proc_lsst.multi:170 [INFO] [multi] job_id 16896441 proc_lsst.multi INFO: [multi] job_id 16896441 2024-03-07 15:07:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 15:07:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:06 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896441 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896441 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 16896441 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-07 15:07:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-07 15:07:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 15:07:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3324 try 0 launched on executor multi with executor id 3325 parsl.dataflow.dflow INFO: Standard output for task 3324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/5bcaa6fa-d146-4f8b-8968-f8dbed0f7feb_isrForCrosstalkSources_1125946_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/5bcaa6fa-d146-4f8b-8968-f8dbed0f7feb_isrForCrosstalkSources_1125946_2.stderr parsl.dataflow.dflow DEBUG: Task 3325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3325 has memoization hash 79cda1b6b69e43f0b24dec637b9ef47d parsl.dataflow.memoization INFO: Task 3325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ee520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3325 try 0 launched on executor multi with executor id 3326 parsl.dataflow.dflow INFO: Standard output for task 3325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/8e5c5d40-a348-4923-b0e7-aa7dced57321_isrForCrosstalkSources_1125918_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/8e5c5d40-a348-4923-b0e7-aa7dced57321_isrForCrosstalkSources_1125918_26.stderr parsl.dataflow.dflow DEBUG: Task 3326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3326 has memoization hash 3827cd42fde5fd1740d6d738c0a84ef5 parsl.dataflow.memoization INFO: Task 3326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ec2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3326 try 0 launched on executor multi with executor id 3327 parsl.dataflow.dflow INFO: Standard output for task 3326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/131461e2-269c-432b-9a30-de59af3fa303_isrForCrosstalkSources_1125895_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/131461e2-269c-432b-9a30-de59af3fa303_isrForCrosstalkSources_1125895_13.stderr parsl.dataflow.dflow DEBUG: Task 3327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3327 has memoization hash 3a6e654cab9e080c222f3a3ca701e8ef parsl.dataflow.memoization INFO: Task 3327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ec220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3327 try 0 launched on executor multi with executor id 3328 parsl.dataflow.dflow INFO: Standard output for task 3327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/b11f8f93-7dc9-4c4c-9e94-318a01fdf9d6_isrForCrosstalkSources_1125907_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/b11f8f93-7dc9-4c4c-9e94-318a01fdf9d6_isrForCrosstalkSources_1125907_58.stderr parsl.dataflow.dflow DEBUG: Task 3328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3328 has memoization hash ac89163cb71786fa989afe6541b4e69a parsl.dataflow.memoization INFO: Task 3328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ed760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3328 try 0 launched on executor multi with executor id 3329 parsl.dataflow.dflow INFO: Standard output for task 3328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/4e22bed5-a71e-405d-bb0e-ee096d645966_isrForCrosstalkSources_1125896_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/4e22bed5-a71e-405d-bb0e-ee096d645966_isrForCrosstalkSources_1125896_28.stderr parsl.dataflow.dflow DEBUG: Task 3329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3329 has memoization hash e2e633b97fadc180fe9d7d81acba1d66 parsl.dataflow.memoization INFO: Task 3329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ec0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3329 try 0 launched on executor multi with executor id 3330 parsl.dataflow.dflow INFO: Standard output for task 3329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/9845aac7-1156-46a1-87ec-f1e7367a6738_isrForCrosstalkSources_1125938_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/9845aac7-1156-46a1-87ec-f1e7367a6738_isrForCrosstalkSources_1125938_15.stderr parsl.dataflow.dflow DEBUG: Task 3330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3330 has memoization hash 562565d1a39ca7915192a228aff30dc6 parsl.dataflow.memoization INFO: Task 3330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39284b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3330 try 0 launched on executor multi with executor id 3331 parsl.dataflow.dflow INFO: Standard output for task 3330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/7763cbbf-0627-4571-adc1-3534d280d0c0_isrForCrosstalkSources_1125926_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/7763cbbf-0627-4571-adc1-3534d280d0c0_isrForCrosstalkSources_1125926_4.stderr parsl.dataflow.dflow DEBUG: Task 3331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3331 has memoization hash b006955b39497bd458cf12f1ff4b9bd3 parsl.dataflow.memoization INFO: Task 3331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a391899e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3331 try 0 launched on executor multi with executor id 3332 parsl.dataflow.dflow INFO: Standard output for task 3331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/42e64dae-a06a-48bb-be56-44b14c20f686_isrForCrosstalkSources_1125897_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/42e64dae-a06a-48bb-be56-44b14c20f686_isrForCrosstalkSources_1125897_8.stderr parsl.dataflow.dflow DEBUG: Task 3332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3332 has memoization hash 5b984a6a0476be8d946d47de45bb8b5f parsl.dataflow.memoization INFO: Task 3332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ef7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3332 try 0 launched on executor multi with executor id 3333 parsl.dataflow.dflow INFO: Standard output for task 3332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/90eca44a-b19d-4061-988c-073eb2b8d947_isrForCrosstalkSources_1125935_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/90eca44a-b19d-4061-988c-073eb2b8d947_isrForCrosstalkSources_1125935_14.stderr parsl.dataflow.dflow DEBUG: Task 3333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3333 has memoization hash 6d0334b5c26d305e043f1ed05bd9befc parsl.dataflow.memoization INFO: Task 3333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3333 try 0 launched on executor multi with executor id 3334 parsl.dataflow.dflow INFO: Standard output for task 3333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/ef0ed036-dfd4-4a94-ae8b-aa906d056796_isrForCrosstalkSources_1125912_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/ef0ed036-dfd4-4a94-ae8b-aa906d056796_isrForCrosstalkSources_1125912_52.stderr parsl.dataflow.dflow DEBUG: Task 3334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3334 has memoization hash 2007a08db30168a603c196a6a3fc1470 parsl.dataflow.memoization INFO: Task 3334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3334 try 0 launched on executor multi with executor id 3335 parsl.dataflow.dflow INFO: Standard output for task 3334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/22f4aeda-60f9-449f-9a4c-718f59f3f776_isrForCrosstalkSources_1125912_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/22f4aeda-60f9-449f-9a4c-718f59f3f776_isrForCrosstalkSources_1125912_2.stderr parsl.dataflow.dflow DEBUG: Task 3335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3335 has memoization hash 6ada104746242e25500ae5136634e6f6 parsl.dataflow.memoization INFO: Task 3335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a391898a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3335 try 0 launched on executor multi with executor id 3336 parsl.dataflow.dflow INFO: Standard output for task 3335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/96099489-d02e-4e1f-b8ec-13be595fab5b_isrForCrosstalkSources_1125901_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/96099489-d02e-4e1f-b8ec-13be595fab5b_isrForCrosstalkSources_1125901_29.stderr parsl.dataflow.dflow DEBUG: Task 3336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3336 has memoization hash fc036619aa5409555f2e3554e15cc84a parsl.dataflow.memoization INFO: Task 3336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ef560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3336 try 0 launched on executor multi with executor id 3337 parsl.dataflow.dflow INFO: Standard output for task 3336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/7df06520-931d-4686-b88a-20846aefe82a_isrForCrosstalkSources_1125941_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/7df06520-931d-4686-b88a-20846aefe82a_isrForCrosstalkSources_1125941_26.stderr parsl.dataflow.dflow DEBUG: Task 3337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3337 has memoization hash 67fc854ed095524a3fba4e3eee213d5b parsl.dataflow.memoization INFO: Task 3337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ed620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3337 try 0 launched on executor multi with executor id 3338 parsl.dataflow.dflow INFO: Standard output for task 3337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a2c879f4-0837-4d31-b8b1-b588475452ca_isrForCrosstalkSources_1125933_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a2c879f4-0837-4d31-b8b1-b588475452ca_isrForCrosstalkSources_1125933_2.stderr parsl.dataflow.dflow DEBUG: Task 3338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3338 has memoization hash f96e020d72e21bcdafd56fb36f240b3e parsl.dataflow.memoization INFO: Task 3338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ef420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3338 try 0 launched on executor multi with executor id 3339 parsl.dataflow.dflow INFO: Standard output for task 3338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/cd9f2a79-9a23-469d-945c-c5541e3e4765_isrForCrosstalkSources_1125935_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/cd9f2a79-9a23-469d-945c-c5541e3e4765_isrForCrosstalkSources_1125935_15.stderr parsl.dataflow.dflow DEBUG: Task 3339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3339 has memoization hash 423f27c556c71d4034fb73e17eb6f93b parsl.dataflow.memoization INFO: Task 3339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ef2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3339 try 0 launched on executor multi with executor id 3340 parsl.dataflow.dflow INFO: Standard output for task 3339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/9b080fb4-2eda-4de1-98c0-f9ce150aad58_isrForCrosstalkSources_1125894_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/9b080fb4-2eda-4de1-98c0-f9ce150aad58_isrForCrosstalkSources_1125894_46.stderr parsl.dataflow.dflow DEBUG: Task 3340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3340 has memoization hash 4ffc30607f98506d193c4bb401ee0ee1 parsl.dataflow.memoization INFO: Task 3340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ef380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3340 try 0 launched on executor multi with executor id 3341 parsl.dataflow.dflow INFO: Standard output for task 3340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/38fd74cc-db59-4505-86e7-863b337e1451_isrForCrosstalkSources_1125937_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/38fd74cc-db59-4505-86e7-863b337e1451_isrForCrosstalkSources_1125937_27.stderr parsl.dataflow.dflow DEBUG: Task 3341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3341 has memoization hash f03aaa21f55c6e393abbf7e3bbff420f parsl.dataflow.memoization INFO: Task 3341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ef4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3341 try 0 launched on executor multi with executor id 3342 parsl.dataflow.dflow INFO: Standard output for task 3341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e52cd6db-9854-49b6-b8d6-cfab6ded6e9e_isrForCrosstalkSources_1125901_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e52cd6db-9854-49b6-b8d6-cfab6ded6e9e_isrForCrosstalkSources_1125901_51.stderr parsl.dataflow.dflow DEBUG: Task 3342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3342 has memoization hash 7b492faba184ba16e59df8ac064fd502 parsl.dataflow.memoization INFO: Task 3342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ef240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3342 try 0 launched on executor multi with executor id 3343 parsl.dataflow.dflow INFO: Standard output for task 3342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/29e462be-b420-4386-b714-e23ab146404e_isrForCrosstalkSources_1125900_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/29e462be-b420-4386-b714-e23ab146404e_isrForCrosstalkSources_1125900_27.stderr parsl.dataflow.dflow DEBUG: Task 3343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3343 has memoization hash 0258a081329bfc9beaab4e79802367a2 parsl.dataflow.memoization INFO: Task 3343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ef100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3343 try 0 launched on executor multi with executor id 3344 parsl.dataflow.dflow INFO: Standard output for task 3343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/6fe5fe20-3456-4a3a-afa7-50c2a7a66a9e_isrForCrosstalkSources_1125918_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/6fe5fe20-3456-4a3a-afa7-50c2a7a66a9e_isrForCrosstalkSources_1125918_20.stderr parsl.dataflow.dflow DEBUG: Task 3344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3344 has memoization hash 07c60bc68dd6d86d2958515327bc17b0 parsl.dataflow.memoization INFO: Task 3344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ef1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3344 try 0 launched on executor multi with executor id 3345 parsl.dataflow.dflow INFO: Standard output for task 3344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/38ec5954-8b7d-45cb-8b10-0ec4c85ace28_isrForCrosstalkSources_1125926_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/38ec5954-8b7d-45cb-8b10-0ec4c85ace28_isrForCrosstalkSources_1125926_24.stderr parsl.dataflow.dflow DEBUG: Task 3345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3345 has memoization hash 526bfe49d77a5cefcfcc482d0e08936f parsl.dataflow.memoization INFO: Task 3345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392eefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3345 try 0 launched on executor multi with executor id 3346 parsl.dataflow.dflow INFO: Standard output for task 3345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/8bd804d3-bf0b-40a1-ab1d-3f34cabe9763_isrForCrosstalkSources_1125920_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/8bd804d3-bf0b-40a1-ab1d-3f34cabe9763_isrForCrosstalkSources_1125920_59.stderr parsl.dataflow.dflow DEBUG: Task 3346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3346 has memoization hash 00cc3bfe88257fc6b38cfa2834af3465 parsl.dataflow.memoization INFO: Task 3346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392eef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3346 try 0 launched on executor multi with executor id 3347 parsl.dataflow.dflow INFO: Standard output for task 3346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/5595126d-26e1-44a6-90a0-eabee85ff750_isrForCrosstalkSources_1125928_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/5595126d-26e1-44a6-90a0-eabee85ff750_isrForCrosstalkSources_1125928_56.stderr parsl.dataflow.dflow DEBUG: Task 3347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3347 has memoization hash da21ee426447f5159cda11e4a11b2186 parsl.dataflow.memoization INFO: Task 3347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ef060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3347 try 0 launched on executor multi with executor id 3348 parsl.dataflow.dflow INFO: Standard output for task 3347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/36d3dc80-913a-4257-b2a9-a546499da9d0_isrForCrosstalkSources_1125919_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/36d3dc80-913a-4257-b2a9-a546499da9d0_isrForCrosstalkSources_1125919_47.stderr parsl.dataflow.dflow DEBUG: Task 3348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3348 has memoization hash 00489d3dc30c31d07f474b4293f90b5d parsl.dataflow.memoization INFO: Task 3348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392eede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3348 try 0 launched on executor multi with executor id 3349 parsl.dataflow.dflow INFO: Standard output for task 3348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/90068e94-18a8-4ed1-930c-c77f2d0b65ec_isrForCrosstalkSources_1125907_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/90068e94-18a8-4ed1-930c-c77f2d0b65ec_isrForCrosstalkSources_1125907_4.stderr parsl.dataflow.dflow DEBUG: Task 3349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3349 has memoization hash 17738a4881976039da0860511df867e3 parsl.dataflow.memoization INFO: Task 3349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392eee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3349 try 0 launched on executor multi with executor id 3350 parsl.dataflow.dflow INFO: Standard output for task 3349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/4a130c25-d530-4d62-afc6-9d6ed7125179_isrForCrosstalkSources_1125915_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/4a130c25-d530-4d62-afc6-9d6ed7125179_isrForCrosstalkSources_1125915_57.stderr parsl.dataflow.dflow DEBUG: Task 3350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3350 has memoization hash c46fe4fd97089e18c990e132fa9b3c41 parsl.dataflow.memoization INFO: Task 3350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392eed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3350 try 0 launched on executor multi with executor id 3351 parsl.dataflow.dflow INFO: Standard output for task 3350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/08c84218-3932-4a12-831d-38176efeec10_isrForCrosstalkSources_1125910_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/08c84218-3932-4a12-831d-38176efeec10_isrForCrosstalkSources_1125910_36.stderr parsl.dataflow.dflow DEBUG: Task 3351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3351 has memoization hash cc7458520bb3ce77f6178b8d96cd24ca parsl.dataflow.memoization INFO: Task 3351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392eec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3351 try 0 launched on executor multi with executor id 3352 parsl.dataflow.dflow INFO: Standard output for task 3351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a9b8b3ea-9e20-461d-8a24-68d8ed0d8b91_isrForCrosstalkSources_1125938_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a9b8b3ea-9e20-461d-8a24-68d8ed0d8b91_isrForCrosstalkSources_1125938_8.stderr parsl.dataflow.dflow DEBUG: Task 3352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3352 has memoization hash ea45dd870315fb9e80a39fc8c2294d50 parsl.dataflow.memoization INFO: Task 3352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392eeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3352 try 0 launched on executor multi with executor id 3353 parsl.dataflow.dflow INFO: Standard output for task 3352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/648cbc32-be57-43b0-a205-13c652989f12_isrForCrosstalkSources_1125915_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/648cbc32-be57-43b0-a205-13c652989f12_isrForCrosstalkSources_1125915_17.stderr parsl.dataflow.dflow DEBUG: Task 3353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3353 has memoization hash 0e6bb232a5e4bde232e01cdc67307e87 parsl.dataflow.memoization INFO: Task 3353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392eeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3353 try 0 launched on executor multi with executor id 3354 parsl.dataflow.dflow INFO: Standard output for task 3353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/6eb105f4-d91e-4e40-bc4b-2b4583228a94_isrForCrosstalkSources_1125906_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/6eb105f4-d91e-4e40-bc4b-2b4583228a94_isrForCrosstalkSources_1125906_49.stderr parsl.dataflow.dflow DEBUG: Task 3354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3354 has memoization hash 6f6be80d1f726b1f3874fd82eb70fe18 parsl.dataflow.memoization INFO: Task 3354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392eeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3354 try 0 launched on executor multi with executor id 3355 parsl.dataflow.dflow INFO: Standard output for task 3354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/89584e9e-3b70-48f8-83cd-31aa81e834d0_isrForCrosstalkSources_1125923_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/89584e9e-3b70-48f8-83cd-31aa81e834d0_isrForCrosstalkSources_1125923_34.stderr parsl.dataflow.dflow DEBUG: Task 3355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3355 has memoization hash 856c5f03e2ad52612d178edb1e98bc53 parsl.dataflow.memoization INFO: Task 3355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392eea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3355 try 0 launched on executor multi with executor id 3356 parsl.dataflow.dflow INFO: Standard output for task 3355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/61f21116-753f-4327-8c45-5a0349bf2367_isrForCrosstalkSources_1125935_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/61f21116-753f-4327-8c45-5a0349bf2367_isrForCrosstalkSources_1125935_33.stderr parsl.dataflow.dflow DEBUG: Task 3356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3356 has memoization hash 61df988d20be96e2a011ee1f4efe47ab parsl.dataflow.memoization INFO: Task 3356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ee8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3356 try 0 launched on executor multi with executor id 3357 parsl.dataflow.dflow INFO: Standard output for task 3356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3852d808-f64b-437d-bc14-04bf3c8b20c2_isrForCrosstalkSources_1125920_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3852d808-f64b-437d-bc14-04bf3c8b20c2_isrForCrosstalkSources_1125920_49.stderr parsl.dataflow.dflow DEBUG: Task 3357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3357 has memoization hash 3563a0d9bec8228d610fb9dd08ab0e08 parsl.dataflow.memoization INFO: Task 3357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ee840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3357 try 0 launched on executor multi with executor id 3358 parsl.dataflow.dflow INFO: Standard output for task 3357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a4aa1150-569d-47a3-b90a-c59be1327f78_isrForCrosstalkSources_1125921_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a4aa1150-569d-47a3-b90a-c59be1327f78_isrForCrosstalkSources_1125921_2.stderr parsl.dataflow.dflow DEBUG: Task 3358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3358 has memoization hash 210904dc47d9dcd24ff5b0e1bebfb7f6 parsl.dataflow.memoization INFO: Task 3358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ee980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3358 try 0 launched on executor multi with executor id 3359 parsl.dataflow.dflow INFO: Standard output for task 3358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/0c9e94d3-c97d-4ffb-97e0-b966078f7ce7_isrForCrosstalkSources_1125920_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/0c9e94d3-c97d-4ffb-97e0-b966078f7ce7_isrForCrosstalkSources_1125920_42.stderr parsl.dataflow.dflow DEBUG: Task 3359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3359 has memoization hash 2993549bc291e1ee180f22ab8477c7d6 parsl.dataflow.memoization INFO: Task 3359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ee7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3359 try 0 launched on executor multi with executor id 3360 parsl.dataflow.dflow INFO: Standard output for task 3359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/e63112ed-724d-4bc4-b316-1c373e177e68_isrForCrosstalkSources_1125918_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/e63112ed-724d-4bc4-b316-1c373e177e68_isrForCrosstalkSources_1125918_22.stderr parsl.dataflow.dflow DEBUG: Task 3360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3360 has memoization hash 89fc7529b9705ceba97bd239a68dc2ac parsl.dataflow.memoization INFO: Task 3360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ee660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3360 try 0 launched on executor multi with executor id 3361 parsl.dataflow.dflow INFO: Standard output for task 3360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/9b6eaacd-42a3-4f94-a1f5-dbea09627ffc_isrForCrosstalkSources_1125942_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/9b6eaacd-42a3-4f94-a1f5-dbea09627ffc_isrForCrosstalkSources_1125942_51.stderr parsl.dataflow.dflow DEBUG: Task 3361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3361 has memoization hash 3f9ea73cfb6065a70c9ef53172379289 parsl.dataflow.memoization INFO: Task 3361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ee700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3361 try 0 launched on executor multi with executor id 3362 parsl.dataflow.dflow INFO: Standard output for task 3361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/db004396-dec6-4381-a301-4bbc6d114088_isrForCrosstalkSources_1125926_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/db004396-dec6-4381-a301-4bbc6d114088_isrForCrosstalkSources_1125926_27.stderr parsl.dataflow.dflow DEBUG: Task 3362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3362 has memoization hash 93041748238e31f3ff721c5092f9d138 parsl.dataflow.memoization INFO: Task 3362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ee5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3362 try 0 launched on executor multi with executor id 3363 parsl.dataflow.dflow INFO: Standard output for task 3362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/7290d2a9-f378-4b1a-ac5c-107a4f1583d9_isrForCrosstalkSources_1125890_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/7290d2a9-f378-4b1a-ac5c-107a4f1583d9_isrForCrosstalkSources_1125890_34.stderr parsl.dataflow.dflow DEBUG: Task 3363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3363 has memoization hash 6ae1f31b035356b2e4aa9963b5de508f parsl.dataflow.memoization INFO: Task 3363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ee480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3363 try 0 launched on executor multi with executor id 3364 parsl.dataflow.dflow INFO: Standard output for task 3363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/8cb30c4b-b49f-431c-9602-c1726d49c3c6_isrForCrosstalkSources_1125937_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/8cb30c4b-b49f-431c-9602-c1726d49c3c6_isrForCrosstalkSources_1125937_6.stderr parsl.dataflow.dflow DEBUG: Task 3364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3364 has memoization hash 449afdc85a20ccf8efc19e76096853b5 parsl.dataflow.memoization INFO: Task 3364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ed580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3364 try 0 launched on executor multi with executor id 3365 parsl.dataflow.dflow INFO: Standard output for task 3364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/2a1b1e74-0831-44f1-95cc-43c8e900753a_isrForCrosstalkSources_1125928_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/2a1b1e74-0831-44f1-95cc-43c8e900753a_isrForCrosstalkSources_1125928_37.stderr parsl.dataflow.dflow DEBUG: Task 3365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3365 has memoization hash 653963b4f41dde81b92e049112b70421 parsl.dataflow.memoization INFO: Task 3365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392efa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3365 try 0 launched on executor multi with executor id 3366 parsl.dataflow.dflow INFO: Standard output for task 3365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/36260e67-f761-4c3b-90e5-631abebcfaea_isrForCrosstalkSources_1125896_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/36260e67-f761-4c3b-90e5-631abebcfaea_isrForCrosstalkSources_1125896_39.stderr parsl.dataflow.dflow DEBUG: Task 3366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3366 has memoization hash fb3f94dd27a7940b641177ba17736c5d parsl.dataflow.memoization INFO: Task 3366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ef9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3366 try 0 launched on executor multi with executor id 3367 parsl.dataflow.dflow INFO: Standard output for task 3366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/048f17dd-5a91-4222-ae88-e5cb626d0bc0_isrForCrosstalkSources_1125893_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/048f17dd-5a91-4222-ae88-e5cb626d0bc0_isrForCrosstalkSources_1125893_19.stderr parsl.dataflow.dflow DEBUG: Task 3367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3367 has memoization hash 02bf6420973e6a1a40fb3ab38788f06f parsl.dataflow.memoization INFO: Task 3367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392efb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3367 try 0 launched on executor multi with executor id 3368 parsl.dataflow.dflow INFO: Standard output for task 3367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/146ef217-2f11-4b2d-a2e4-7bd1600f4365_isrForCrosstalkSources_1125933_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/146ef217-2f11-4b2d-a2e4-7bd1600f4365_isrForCrosstalkSources_1125933_53.stderr parsl.dataflow.dflow DEBUG: Task 3368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3368 has memoization hash 71e106329d4d760786e71e61eaf26252 parsl.dataflow.memoization INFO: Task 3368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392efba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3368 try 0 launched on executor multi with executor id 3369 parsl.dataflow.dflow INFO: Standard output for task 3368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/53b68ef4-254f-4fc6-8f61-b87ab9e8fe8e_isrForCrosstalkSources_1125924_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/53b68ef4-254f-4fc6-8f61-b87ab9e8fe8e_isrForCrosstalkSources_1125924_3.stderr parsl.dataflow.dflow DEBUG: Task 3369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3369 has memoization hash fff33a3392f1e5e695bd870d508eacdd parsl.dataflow.memoization INFO: Task 3369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392efc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3369 try 0 launched on executor multi with executor id 3370 parsl.dataflow.dflow INFO: Standard output for task 3369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/07cc0e37-3749-428d-8ed8-c28d89fe286c_isrForCrosstalkSources_1125928_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/07cc0e37-3749-428d-8ed8-c28d89fe286c_isrForCrosstalkSources_1125928_9.stderr parsl.dataflow.dflow DEBUG: Task 3370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3370 has memoization hash 92d28894775e3caffa36eaf23e83e2ff parsl.dataflow.memoization INFO: Task 3370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392efd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3370 try 0 launched on executor multi with executor id 3371 parsl.dataflow.dflow INFO: Standard output for task 3370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/84a670d0-d21d-40b0-92fa-f70abae23737_isrForCrosstalkSources_1125935_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/84a670d0-d21d-40b0-92fa-f70abae23737_isrForCrosstalkSources_1125935_46.stderr parsl.dataflow.dflow DEBUG: Task 3371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3371 has memoization hash d1694ca0c26962eeb2d39b10bb8ac1ce parsl.dataflow.memoization INFO: Task 3371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392efce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3371 try 0 launched on executor multi with executor id 3372 parsl.dataflow.dflow INFO: Standard output for task 3371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/23f41d5c-1fdb-4b88-8462-164cafb3c090_isrForCrosstalkSources_1125943_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/23f41d5c-1fdb-4b88-8462-164cafb3c090_isrForCrosstalkSources_1125943_43.stderr parsl.dataflow.dflow DEBUG: Task 3372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3372 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3372 has memoization hash 707275198cfb9cdb798464b4284c1d6e parsl.dataflow.memoization INFO: Task 3372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392efec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3372 try 0 launched on executor multi with executor id 3373 parsl.dataflow.dflow INFO: Standard output for task 3372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/f9bf5974-484f-4a43-8052-7240e1dfca85_isrForCrosstalkSources_1125907_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/f9bf5974-484f-4a43-8052-7240e1dfca85_isrForCrosstalkSources_1125907_19.stderr parsl.dataflow.dflow DEBUG: Task 3373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3373 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3373 has memoization hash ab4c3b13e3fa3c458a93fb5d481fcba4 parsl.dataflow.memoization INFO: Task 3373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392efe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3373 try 0 launched on executor multi with executor id 3374 parsl.dataflow.dflow INFO: Standard output for task 3373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/b05fbf42-bae0-4225-889e-a7deaa8f8b89_isrForCrosstalkSources_1125914_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/b05fbf42-bae0-4225-889e-a7deaa8f8b89_isrForCrosstalkSources_1125914_30.stderr parsl.dataflow.dflow DEBUG: Task 3374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3374 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3374 has memoization hash 6260dd5b8509d624f2b2e97502d9b95a parsl.dataflow.memoization INFO: Task 3374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ef600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3374 try 0 launched on executor multi with executor id 3375 parsl.dataflow.dflow INFO: Standard output for task 3374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/f7b72e68-e70d-4524-8ab4-6aa62542137f_isrForCrosstalkSources_1125914_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/f7b72e68-e70d-4524-8ab4-6aa62542137f_isrForCrosstalkSources_1125914_5.stderr parsl.dataflow.dflow DEBUG: Task 3375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3375 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3375 has memoization hash 6401b36d3b03a85c7feb5e7b6509e413 parsl.dataflow.memoization INFO: Task 3375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:07 proc_lsst.multi:170 [INFO] [multi] job_id 16896442 proc_lsst.multi INFO: [multi] job_id 16896442 2024-03-07 15:07:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 15:07:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:07 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896442 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896442 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 16896442 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-07 15:07:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-07 15:07:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 15:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3375 try 0 launched on executor multi with executor id 3376 parsl.dataflow.dflow INFO: Standard output for task 3375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/329d093f-04a7-494f-93d8-be2966069258_isrForCrosstalkSources_1125893_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/329d093f-04a7-494f-93d8-be2966069258_isrForCrosstalkSources_1125893_58.stderr parsl.dataflow.dflow DEBUG: Task 3376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3376 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3376 has memoization hash 288156ed677292e23e0c49bb78e9938c parsl.dataflow.memoization INFO: Task 3376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ee3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3376 try 0 launched on executor multi with executor id 3377 parsl.dataflow.dflow INFO: Standard output for task 3376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/d2c5c3fd-b48d-4768-be1e-7bd6eedc416b_isrForCrosstalkSources_1125929_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/d2c5c3fd-b48d-4768-be1e-7bd6eedc416b_isrForCrosstalkSources_1125929_56.stderr parsl.dataflow.dflow DEBUG: Task 3377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3377 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3377 has memoization hash dd9fc261c0664989289e9e368fe2022c parsl.dataflow.memoization INFO: Task 3377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392eff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3377 try 0 launched on executor multi with executor id 3378 parsl.dataflow.dflow INFO: Standard output for task 3377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/468d8e8b-0d45-4d6b-8f54-d2ca29e7b17d_isrForCrosstalkSources_1125926_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/468d8e8b-0d45-4d6b-8f54-d2ca29e7b17d_isrForCrosstalkSources_1125926_45.stderr parsl.dataflow.dflow DEBUG: Task 3378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3378 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3378 has memoization hash 9f8c5b841b67af6dc818ab4e063f6b23 parsl.dataflow.memoization INFO: Task 3378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a392ef740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3378 try 0 launched on executor multi with executor id 3379 parsl.dataflow.dflow INFO: Standard output for task 3378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/5580be57-63ac-44b9-959e-d30f85cef70b_isrForCrosstalkSources_1125940_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/5580be57-63ac-44b9-959e-d30f85cef70b_isrForCrosstalkSources_1125940_53.stderr parsl.dataflow.dflow DEBUG: Task 3379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3379 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3379 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3379 has memoization hash 29373feb203f0b780441813da2a136dd parsl.dataflow.memoization INFO: Task 3379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3379 try 0 launched on executor multi with executor id 3380 parsl.dataflow.dflow INFO: Standard output for task 3379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2a31e8b6-499c-4566-9d87-1593677471b0_isrForCrosstalkSources_1125933_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2a31e8b6-499c-4566-9d87-1593677471b0_isrForCrosstalkSources_1125933_30.stderr parsl.dataflow.dflow DEBUG: Task 3380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3380 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3380 has memoization hash 2d986a9c733fc05b63f1904027a2f7d4 parsl.dataflow.memoization INFO: Task 3380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3380 try 0 launched on executor multi with executor id 3381 parsl.dataflow.dflow INFO: Standard output for task 3380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/ef2a79fc-a87f-486a-ac24-99ae040db962_isrForCrosstalkSources_1125938_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/ef2a79fc-a87f-486a-ac24-99ae040db962_isrForCrosstalkSources_1125938_12.stderr parsl.dataflow.dflow DEBUG: Task 3381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3381 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3381 has memoization hash 07695cec289a300527cf1384cd158fc1 parsl.dataflow.memoization INFO: Task 3381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a391880e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3381 try 0 launched on executor multi with executor id 3382 parsl.dataflow.dflow INFO: Standard output for task 3381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/884389d2-329a-4a62-83ad-708fd93949be_isrForCrosstalkSources_1125910_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/884389d2-329a-4a62-83ad-708fd93949be_isrForCrosstalkSources_1125910_46.stderr parsl.dataflow.dflow DEBUG: Task 3382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3382 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3382 has memoization hash bbd532f80067fef961306cdc41d1af35 parsl.dataflow.memoization INFO: Task 3382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3382 try 0 launched on executor multi with executor id 3383 parsl.dataflow.dflow INFO: Standard output for task 3382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/87e5b210-b4eb-4b70-b1ce-848b8186a14c_isrForCrosstalkSources_1125918_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/87e5b210-b4eb-4b70-b1ce-848b8186a14c_isrForCrosstalkSources_1125918_60.stderr parsl.dataflow.dflow DEBUG: Task 3383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3383 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3383 has memoization hash 13a4b4880d2d4384651538f939043caa parsl.dataflow.memoization INFO: Task 3383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3383 try 0 launched on executor multi with executor id 3384 parsl.dataflow.dflow INFO: Standard output for task 3383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/c6fa70e2-1b6f-4cf1-8eed-e5b667519539_isrForCrosstalkSources_1125890_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/c6fa70e2-1b6f-4cf1-8eed-e5b667519539_isrForCrosstalkSources_1125890_1.stderr parsl.dataflow.dflow DEBUG: Task 3384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3384 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3384 has memoization hash fe2d6b9cb85db4140b2b8b434d7e9fca parsl.dataflow.memoization INFO: Task 3384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3384 try 0 launched on executor multi with executor id 3385 parsl.dataflow.dflow INFO: Standard output for task 3384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/8c624f4b-9760-4568-8f7c-fb423ada9194_isrForCrosstalkSources_1125907_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/8c624f4b-9760-4568-8f7c-fb423ada9194_isrForCrosstalkSources_1125907_28.stderr parsl.dataflow.dflow DEBUG: Task 3385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3385 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3385 has memoization hash 355278652aa2781402496b112a0edda0 parsl.dataflow.memoization INFO: Task 3385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3385 try 0 launched on executor multi with executor id 3386 parsl.dataflow.dflow INFO: Standard output for task 3385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/f482997d-6d8a-48ce-8a22-fc150abd564b_isrForCrosstalkSources_1125938_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/f482997d-6d8a-48ce-8a22-fc150abd564b_isrForCrosstalkSources_1125938_51.stderr parsl.dataflow.dflow DEBUG: Task 3386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3386 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3386 has memoization hash fc577ae6fef4c748bbcadb9249a2e670 parsl.dataflow.memoization INFO: Task 3386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3386 try 0 launched on executor multi with executor id 3387 parsl.dataflow.dflow INFO: Standard output for task 3386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8e6350a9-1ec1-40fa-93d7-cc83dbd83504_isrForCrosstalkSources_1125936_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8e6350a9-1ec1-40fa-93d7-cc83dbd83504_isrForCrosstalkSources_1125936_33.stderr parsl.dataflow.dflow DEBUG: Task 3387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3387 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3387 has memoization hash 69f47d2bf1cf4160ee1ca5cc0318cd72 parsl.dataflow.memoization INFO: Task 3387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3387 try 0 launched on executor multi with executor id 3388 parsl.dataflow.dflow INFO: Standard output for task 3387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/a2f482b0-8c8a-4ac5-b6b9-2a6df27a86a8_isrForCrosstalkSources_1125908_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/a2f482b0-8c8a-4ac5-b6b9-2a6df27a86a8_isrForCrosstalkSources_1125908_5.stderr parsl.dataflow.dflow DEBUG: Task 3388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3388 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3388 has memoization hash 5d19873cfbbcfcda040a8dfdf4b7333e parsl.dataflow.memoization INFO: Task 3388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3388 try 0 launched on executor multi with executor id 3389 parsl.dataflow.dflow INFO: Standard output for task 3388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/d799f247-2a82-469d-b030-dbdd79cf47dc_isrForCrosstalkSources_1125947_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/d799f247-2a82-469d-b030-dbdd79cf47dc_isrForCrosstalkSources_1125947_39.stderr parsl.dataflow.dflow DEBUG: Task 3389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3389 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3389 has memoization hash ee848bc1774fd15d94133d90f910e250 parsl.dataflow.memoization INFO: Task 3389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a391884a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3389 try 0 launched on executor multi with executor id 3390 parsl.dataflow.dflow INFO: Standard output for task 3389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c971b011-a26c-45dd-ba44-945a85f38de2_isrForCrosstalkSources_1125916_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c971b011-a26c-45dd-ba44-945a85f38de2_isrForCrosstalkSources_1125916_28.stderr parsl.dataflow.dflow DEBUG: Task 3390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3390 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3390 has memoization hash 300cbaf00c8054ceee2abf36153aa420 parsl.dataflow.memoization INFO: Task 3390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a391882c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3390 try 0 launched on executor multi with executor id 3391 parsl.dataflow.dflow INFO: Standard output for task 3390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/1f36cde0-3f0e-481d-9129-5976dd5d9c0e_isrForCrosstalkSources_1125918_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/1f36cde0-3f0e-481d-9129-5976dd5d9c0e_isrForCrosstalkSources_1125918_59.stderr parsl.dataflow.dflow DEBUG: Task 3391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3391 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3391 has memoization hash 912c449e414442b1ed73364996ba08f6 parsl.dataflow.memoization INFO: Task 3391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3391 try 0 launched on executor multi with executor id 3392 parsl.dataflow.dflow INFO: Standard output for task 3391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/1715b1ce-1365-4645-964d-8ab8db660ad4_isrForCrosstalkSources_1125921_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/1715b1ce-1365-4645-964d-8ab8db660ad4_isrForCrosstalkSources_1125921_48.stderr parsl.dataflow.dflow DEBUG: Task 3392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3392 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3392 has memoization hash 0f9fb023dd371df20c0c31c0490b499d parsl.dataflow.memoization INFO: Task 3392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a391885e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3392 try 0 launched on executor multi with executor id 3393 parsl.dataflow.dflow INFO: Standard output for task 3392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/31ab40e5-400a-4792-b2c0-ac5591ee4064_isrForCrosstalkSources_1125922_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/31ab40e5-400a-4792-b2c0-ac5591ee4064_isrForCrosstalkSources_1125922_62.stderr parsl.dataflow.dflow DEBUG: Task 3393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3393 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3393 has memoization hash 9ba7ddf8bb70c136056b2a317aa20196 parsl.dataflow.memoization INFO: Task 3393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3393 try 0 launched on executor multi with executor id 3394 parsl.dataflow.dflow INFO: Standard output for task 3393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/d34f27db-d9d9-446a-a6ec-29863863f97d_isrForCrosstalkSources_1125936_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/d34f27db-d9d9-446a-a6ec-29863863f97d_isrForCrosstalkSources_1125936_39.stderr parsl.dataflow.dflow DEBUG: Task 3394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3394 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3394 has memoization hash 219f1ad30bd1a52aee0da3c3ee05791a parsl.dataflow.memoization INFO: Task 3394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3394 try 0 launched on executor multi with executor id 3395 parsl.dataflow.dflow INFO: Standard output for task 3394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a323afe7-d47e-4d1c-92b2-a7b5415233fb_isrForCrosstalkSources_1125909_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a323afe7-d47e-4d1c-92b2-a7b5415233fb_isrForCrosstalkSources_1125909_18.stderr parsl.dataflow.dflow DEBUG: Task 3395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3395 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3395 has memoization hash 8acb4a6c4315d41811c9b5aa070e2a82 parsl.dataflow.memoization INFO: Task 3395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3395 try 0 launched on executor multi with executor id 3396 parsl.dataflow.dflow INFO: Standard output for task 3395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/76e9e203-ea93-4a62-a42a-99cf78d8c9d9_isrForCrosstalkSources_1125927_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/76e9e203-ea93-4a62-a42a-99cf78d8c9d9_isrForCrosstalkSources_1125927_15.stderr parsl.dataflow.dflow DEBUG: Task 3396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3396 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3396 has memoization hash 2648d82d3cc3299821dc845f41ff24e5 parsl.dataflow.memoization INFO: Task 3396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a391894e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3396 try 0 launched on executor multi with executor id 3397 parsl.dataflow.dflow INFO: Standard output for task 3396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/6f54ac21-fcd9-4f37-856a-0ff07051a337_isrForCrosstalkSources_1125920_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/6f54ac21-fcd9-4f37-856a-0ff07051a337_isrForCrosstalkSources_1125920_37.stderr parsl.dataflow.dflow DEBUG: Task 3397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3397 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3397 has memoization hash 650b352c402771d0e2b322e426ac87d1 parsl.dataflow.memoization INFO: Task 3397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3397 try 0 launched on executor multi with executor id 3398 parsl.dataflow.dflow INFO: Standard output for task 3397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/0638a0c1-5543-4c9b-a42b-afaa169cf831_isrForCrosstalkSources_1125911_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/0638a0c1-5543-4c9b-a42b-afaa169cf831_isrForCrosstalkSources_1125911_36.stderr parsl.dataflow.dflow DEBUG: Task 3398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3398 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3398 has memoization hash e09059f3b005c841369fc41fade20a47 parsl.dataflow.memoization INFO: Task 3398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3398 try 0 launched on executor multi with executor id 3399 parsl.dataflow.dflow INFO: Standard output for task 3398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/0726c36f-ae2c-417b-8772-7a6c1bd1d133_isrForCrosstalkSources_1125893_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/0726c36f-ae2c-417b-8772-7a6c1bd1d133_isrForCrosstalkSources_1125893_54.stderr parsl.dataflow.dflow DEBUG: Task 3399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3399 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3399 has memoization hash 5e04a3c5904b019a427f748c00156a66 parsl.dataflow.memoization INFO: Task 3399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a391893a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3399 try 0 launched on executor multi with executor id 3400 parsl.dataflow.dflow INFO: Standard output for task 3399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/c4f06387-9265-405d-97c6-46dda499d9f6_isrForCrosstalkSources_1125934_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/c4f06387-9265-405d-97c6-46dda499d9f6_isrForCrosstalkSources_1125934_48.stderr parsl.dataflow.dflow DEBUG: Task 3400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3400 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3400 has memoization hash 9547e8c7cb3f83e66816df5b702f83f4 parsl.dataflow.memoization INFO: Task 3400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3400 try 0 launched on executor multi with executor id 3401 parsl.dataflow.dflow INFO: Standard output for task 3400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/713ee1ef-5602-481f-ad00-4b7abe577bfd_isrForCrosstalkSources_1125908_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/713ee1ef-5602-481f-ad00-4b7abe577bfd_isrForCrosstalkSources_1125908_30.stderr parsl.dataflow.dflow DEBUG: Task 3401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3401 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3401 has memoization hash 65c3db299ef66cf4d8e7111d8f76b536 parsl.dataflow.memoization INFO: Task 3401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a391891c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3401 try 0 launched on executor multi with executor id 3402 parsl.dataflow.dflow INFO: Standard output for task 3401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/5c1063c7-e9a7-4d61-8011-5e693e77f1f4_isrForCrosstalkSources_1125936_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/5c1063c7-e9a7-4d61-8011-5e693e77f1f4_isrForCrosstalkSources_1125936_7.stderr parsl.dataflow.dflow DEBUG: Task 3402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3402 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3402 has memoization hash 7a6fcf49fc68ba0ca05bda8550b5b963 parsl.dataflow.memoization INFO: Task 3402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3402 try 0 launched on executor multi with executor id 3403 parsl.dataflow.dflow INFO: Standard output for task 3402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/a19b3a42-1827-4832-b857-cb3835389b59_isrForCrosstalkSources_1125946_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/a19b3a42-1827-4832-b857-cb3835389b59_isrForCrosstalkSources_1125946_24.stderr parsl.dataflow.dflow DEBUG: Task 3403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3403 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3403 has memoization hash 03798145309629f0bcba64f42b0409c8 parsl.dataflow.memoization INFO: Task 3403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3403 try 0 launched on executor multi with executor id 3404 parsl.dataflow.dflow INFO: Standard output for task 3403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/90d1f4f2-bcdc-4dd4-a94c-f8ae498a4e51_isrForCrosstalkSources_1125939_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/90d1f4f2-bcdc-4dd4-a94c-f8ae498a4e51_isrForCrosstalkSources_1125939_24.stderr parsl.dataflow.dflow DEBUG: Task 3404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3404 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3404 has memoization hash 09cfda5359081358b25117016774c86c parsl.dataflow.memoization INFO: Task 3404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3404 try 0 launched on executor multi with executor id 3405 parsl.dataflow.dflow INFO: Standard output for task 3404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/12eaf46e-7779-4243-ba3c-a60260c62afa_isrForCrosstalkSources_1125932_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/12eaf46e-7779-4243-ba3c-a60260c62afa_isrForCrosstalkSources_1125932_8.stderr parsl.dataflow.dflow DEBUG: Task 3405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3405 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3405 has memoization hash 502ca86e43d96bed9851ecca87025519 parsl.dataflow.memoization INFO: Task 3405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3405 try 0 launched on executor multi with executor id 3406 parsl.dataflow.dflow INFO: Standard output for task 3405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/fa6046bc-e06e-484e-894d-41926ce31a6d_isrForCrosstalkSources_1125935_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/fa6046bc-e06e-484e-894d-41926ce31a6d_isrForCrosstalkSources_1125935_20.stderr parsl.dataflow.dflow DEBUG: Task 3406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3406 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3406 has memoization hash 6737e25449a3c938e50e3712a8608af2 parsl.dataflow.memoization INFO: Task 3406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3406 try 0 launched on executor multi with executor id 3407 parsl.dataflow.dflow INFO: Standard output for task 3406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/5c5a363a-d868-444b-be67-bc38b4d89aab_isrForCrosstalkSources_1125906_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/5c5a363a-d868-444b-be67-bc38b4d89aab_isrForCrosstalkSources_1125906_26.stderr parsl.dataflow.dflow DEBUG: Task 3407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3407 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3407 has memoization hash 6c1268485f415d0942d1fb6f825dc7be parsl.dataflow.memoization INFO: Task 3407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3407 try 0 launched on executor multi with executor id 3408 parsl.dataflow.dflow INFO: Standard output for task 3407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/ab8eab0e-b8e9-4427-a126-9508c3573bb8_isrForCrosstalkSources_1125905_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/ab8eab0e-b8e9-4427-a126-9508c3573bb8_isrForCrosstalkSources_1125905_17.stderr parsl.dataflow.dflow DEBUG: Task 3408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3408 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3408 has memoization hash 8378aa07776820ff49b2ee4c6f84e306 parsl.dataflow.memoization INFO: Task 3408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3408 try 0 launched on executor multi with executor id 3409 parsl.dataflow.dflow INFO: Standard output for task 3408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/1618d355-5c09-4c5b-bff0-f25cb222fa41_isrForCrosstalkSources_1125901_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/1618d355-5c09-4c5b-bff0-f25cb222fa41_isrForCrosstalkSources_1125901_15.stderr parsl.dataflow.dflow DEBUG: Task 3409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3409 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3409 has memoization hash 82a6f46aa0149220ea2fd776eedf950f parsl.dataflow.memoization INFO: Task 3409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3409 try 0 launched on executor multi with executor id 3410 parsl.dataflow.dflow INFO: Standard output for task 3409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/3a1b0051-4859-4be8-b348-4ca805043dd2_isrForCrosstalkSources_1125924_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/3a1b0051-4859-4be8-b348-4ca805043dd2_isrForCrosstalkSources_1125924_26.stderr parsl.dataflow.dflow DEBUG: Task 3410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3410 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3410 has memoization hash 6917c3f9b2342f315ae14313a53ce18d parsl.dataflow.memoization INFO: Task 3410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3410 try 0 launched on executor multi with executor id 3411 parsl.dataflow.dflow INFO: Standard output for task 3410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/304307af-506b-46a4-bee6-850c6fb847fb_isrForCrosstalkSources_1125946_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/304307af-506b-46a4-bee6-850c6fb847fb_isrForCrosstalkSources_1125946_52.stderr parsl.dataflow.dflow DEBUG: Task 3411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3411 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3411 has memoization hash 759a26eb619176fddb727be566299709 parsl.dataflow.memoization INFO: Task 3411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:07 proc_lsst.multi:170 [INFO] [multi] job_id 16896443 proc_lsst.multi INFO: [multi] job_id 16896443 2024-03-07 15:07:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 15:07:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:07 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896443 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896443 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 16896443 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-07 15:07:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-07 15:07:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 15:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3411 try 0 launched on executor multi with executor id 3412 parsl.dataflow.dflow INFO: Standard output for task 3411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/9ebac8d9-dfce-4ab7-9ff0-92231ca0074c_isrForCrosstalkSources_1125915_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/9ebac8d9-dfce-4ab7-9ff0-92231ca0074c_isrForCrosstalkSources_1125915_42.stderr parsl.dataflow.dflow DEBUG: Task 3412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3412 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3412 has memoization hash d78d19859cd655d5f523e3bd1f6987ab parsl.dataflow.memoization INFO: Task 3412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3412 try 0 launched on executor multi with executor id 3413 parsl.dataflow.dflow INFO: Standard output for task 3412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c1b37da5-c009-4158-9408-f6e6e3949244_isrForCrosstalkSources_1125916_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c1b37da5-c009-4158-9408-f6e6e3949244_isrForCrosstalkSources_1125916_7.stderr parsl.dataflow.dflow DEBUG: Task 3413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3413 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3413 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3413 has memoization hash 0b19eb247f9f2ad737b916c50b35caaf parsl.dataflow.memoization INFO: Task 3413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a391896c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3413 try 0 launched on executor multi with executor id 3414 parsl.dataflow.dflow INFO: Standard output for task 3413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a0ac7aa8-18d0-4850-aad1-34315eee3a8d_isrForCrosstalkSources_1125921_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a0ac7aa8-18d0-4850-aad1-34315eee3a8d_isrForCrosstalkSources_1125921_6.stderr parsl.dataflow.dflow DEBUG: Task 3414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3414 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3414 has memoization hash a2f0ba582bebcb828794297b0dc7a5cf parsl.dataflow.memoization INFO: Task 3414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3414 try 0 launched on executor multi with executor id 3415 parsl.dataflow.dflow INFO: Standard output for task 3414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/bd19c350-762b-4f58-9659-eb85b00e2249_isrForCrosstalkSources_1125932_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/bd19c350-762b-4f58-9659-eb85b00e2249_isrForCrosstalkSources_1125932_62.stderr parsl.dataflow.dflow DEBUG: Task 3415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3415 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3415 has memoization hash 4eb62d039aac53fb80b64eae0384ffe9 parsl.dataflow.memoization INFO: Task 3415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3415 try 0 launched on executor multi with executor id 3416 parsl.dataflow.dflow INFO: Standard output for task 3415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/38117664-4ef6-4223-8886-764613d7ec14_isrForCrosstalkSources_1125889_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/38117664-4ef6-4223-8886-764613d7ec14_isrForCrosstalkSources_1125889_28.stderr parsl.dataflow.dflow DEBUG: Task 3416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3416 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3416 has memoization hash 9d2aab99c0fb91d3430430345f4a0182 parsl.dataflow.memoization INFO: Task 3416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3416 try 0 launched on executor multi with executor id 3417 parsl.dataflow.dflow INFO: Standard output for task 3416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/c0fba0ae-5cf5-4da5-ba0f-2bfbbe060fe1_isrForCrosstalkSources_1125938_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/c0fba0ae-5cf5-4da5-ba0f-2bfbbe060fe1_isrForCrosstalkSources_1125938_21.stderr parsl.dataflow.dflow DEBUG: Task 3417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3417 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3417 has memoization hash 06f82f3da1b50b3cb756c27e34b84218 parsl.dataflow.memoization INFO: Task 3417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3417 try 0 launched on executor multi with executor id 3418 parsl.dataflow.dflow INFO: Standard output for task 3417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/10290e67-89c3-4394-bf33-328ddd62a096_isrForCrosstalkSources_1125921_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/10290e67-89c3-4394-bf33-328ddd62a096_isrForCrosstalkSources_1125921_49.stderr parsl.dataflow.dflow DEBUG: Task 3418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3418 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3418 has memoization hash ee5f3515bd0619f818a08eb5bfd03a41 parsl.dataflow.memoization INFO: Task 3418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3418 try 0 launched on executor multi with executor id 3419 parsl.dataflow.dflow INFO: Standard output for task 3418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/44d29fcf-46e9-43b4-9c77-d9c29201bf88_isrForCrosstalkSources_1125924_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/44d29fcf-46e9-43b4-9c77-d9c29201bf88_isrForCrosstalkSources_1125924_39.stderr parsl.dataflow.dflow DEBUG: Task 3419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3419 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3419 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3419 has memoization hash 99784a3231a7f74fdf26aedf394a9117 parsl.dataflow.memoization INFO: Task 3419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a391889a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3419 try 0 launched on executor multi with executor id 3420 parsl.dataflow.dflow INFO: Standard output for task 3419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/bc4773a0-a071-46a3-abff-862f8e1a67f9_isrForCrosstalkSources_1125947_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/bc4773a0-a071-46a3-abff-862f8e1a67f9_isrForCrosstalkSources_1125947_30.stderr parsl.dataflow.dflow DEBUG: Task 3420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3420 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3420 has memoization hash b39863695b8bca8cd548957a1bafd8de parsl.dataflow.memoization INFO: Task 3420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3420 try 0 launched on executor multi with executor id 3421 parsl.dataflow.dflow INFO: Standard output for task 3420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/2472105b-3f16-47b7-b051-4d3e3f7c1f81_isrForCrosstalkSources_1125904_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/2472105b-3f16-47b7-b051-4d3e3f7c1f81_isrForCrosstalkSources_1125904_12.stderr parsl.dataflow.dflow DEBUG: Task 3421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3421 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3421 has memoization hash 16e3978d25a5248bc9f13f134e8f826c parsl.dataflow.memoization INFO: Task 3421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39188680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3421 try 0 launched on executor multi with executor id 3422 parsl.dataflow.dflow INFO: Standard output for task 3421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/91b864ba-c347-45f4-bca0-39912788ed7f_isrForCrosstalkSources_1125917_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/91b864ba-c347-45f4-bca0-39912788ed7f_isrForCrosstalkSources_1125917_41.stderr parsl.dataflow.dflow DEBUG: Task 3422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3422 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3422 has memoization hash bd9bcbe4844be0e95642bef03c2deeca parsl.dataflow.memoization INFO: Task 3422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3422 try 0 launched on executor multi with executor id 3423 parsl.dataflow.dflow INFO: Standard output for task 3422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6cb2de9f-e6c5-497e-8674-1c7e9b47713a_isrForCrosstalkSources_1125927_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6cb2de9f-e6c5-497e-8674-1c7e9b47713a_isrForCrosstalkSources_1125927_4.stderr parsl.dataflow.dflow DEBUG: Task 3423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3423 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3423 has memoization hash cd7c80e9505f670f468bb7d230fc0995 parsl.dataflow.memoization INFO: Task 3423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3423 try 0 launched on executor multi with executor id 3424 parsl.dataflow.dflow INFO: Standard output for task 3423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/af44c822-a9e4-4038-8f17-f2a0ccef9b5c_isrForCrosstalkSources_1125924_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/af44c822-a9e4-4038-8f17-f2a0ccef9b5c_isrForCrosstalkSources_1125924_11.stderr parsl.dataflow.dflow DEBUG: Task 3424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3424 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3424 has memoization hash fc87655f6f6a3a366bb3a22ebee3da46 parsl.dataflow.memoization INFO: Task 3424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3424 try 0 launched on executor multi with executor id 3425 parsl.dataflow.dflow INFO: Standard output for task 3424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/3e7685ea-5ea4-4560-b669-ae9f593a9238_isrForCrosstalkSources_1125924_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/3e7685ea-5ea4-4560-b669-ae9f593a9238_isrForCrosstalkSources_1125924_30.stderr parsl.dataflow.dflow DEBUG: Task 3425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3425 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3425 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3425 has memoization hash 0e8276717368eb4023d4146c33421fc6 parsl.dataflow.memoization INFO: Task 3425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3425 try 0 launched on executor multi with executor id 3426 parsl.dataflow.dflow INFO: Standard output for task 3425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/a4569d9e-3e7b-41ea-b72a-b60c17dc2dd4_isrForCrosstalkSources_1125912_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/a4569d9e-3e7b-41ea-b72a-b60c17dc2dd4_isrForCrosstalkSources_1125912_53.stderr parsl.dataflow.dflow DEBUG: Task 3426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3426 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3426 has memoization hash b147984ef1a7873431348bff7018071c parsl.dataflow.memoization INFO: Task 3426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3426 try 0 launched on executor multi with executor id 3427 parsl.dataflow.dflow INFO: Standard output for task 3426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/4b0f5b93-a29c-49e4-95b1-827b09f2ce1c_isrForCrosstalkSources_1125911_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/4b0f5b93-a29c-49e4-95b1-827b09f2ce1c_isrForCrosstalkSources_1125911_27.stderr parsl.dataflow.dflow DEBUG: Task 3427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3427 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3427 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3427 has memoization hash 892ce461820f4b0a1916351a6a370d1b parsl.dataflow.memoization INFO: Task 3427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3427 try 0 launched on executor multi with executor id 3428 parsl.dataflow.dflow INFO: Standard output for task 3427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f5057a27-d34a-4a89-89fd-3521fce168ce_isrForCrosstalkSources_1125922_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f5057a27-d34a-4a89-89fd-3521fce168ce_isrForCrosstalkSources_1125922_20.stderr parsl.dataflow.dflow DEBUG: Task 3428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3428 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3428 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3428 has memoization hash a1e5f9d9c14437e31c8bad4f663a9d06 parsl.dataflow.memoization INFO: Task 3428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390193a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3428 try 0 launched on executor multi with executor id 3429 parsl.dataflow.dflow INFO: Standard output for task 3428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/5f09e175-ed7c-4fee-9c97-170bac8bddcc_isrForCrosstalkSources_1125942_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/5f09e175-ed7c-4fee-9c97-170bac8bddcc_isrForCrosstalkSources_1125942_8.stderr parsl.dataflow.dflow DEBUG: Task 3429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3429 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3429 has memoization hash dbd8575a03eb3bae4c2e7401a2e620cc parsl.dataflow.memoization INFO: Task 3429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3429 try 0 launched on executor multi with executor id 3430 parsl.dataflow.dflow INFO: Standard output for task 3429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d6934c01-976b-455a-a854-9d3b0002f24f_isrForCrosstalkSources_1125915_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d6934c01-976b-455a-a854-9d3b0002f24f_isrForCrosstalkSources_1125915_31.stderr parsl.dataflow.dflow DEBUG: Task 3430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3430 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3430 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3430 has memoization hash 25989f927fa0cb9dcad5050cb0d478e8 parsl.dataflow.memoization INFO: Task 3430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3430 try 0 launched on executor multi with executor id 3431 parsl.dataflow.dflow INFO: Standard output for task 3430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/e302a956-e170-4586-a124-3fbf110330ed_isrForCrosstalkSources_1125912_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/e302a956-e170-4586-a124-3fbf110330ed_isrForCrosstalkSources_1125912_13.stderr parsl.dataflow.dflow DEBUG: Task 3431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3431 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3431 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3431 has memoization hash eb513c919159743fac8c81405469901c parsl.dataflow.memoization INFO: Task 3431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3431 try 0 launched on executor multi with executor id 3432 parsl.dataflow.dflow INFO: Standard output for task 3431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/8452bea1-b019-4935-b8e2-1a0e3f16738f_isrForCrosstalkSources_1125939_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/8452bea1-b019-4935-b8e2-1a0e3f16738f_isrForCrosstalkSources_1125939_39.stderr parsl.dataflow.dflow DEBUG: Task 3432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3432 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3432 has memoization hash 0fde53d28bf582af139720fe40479ed0 parsl.dataflow.memoization INFO: Task 3432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3432 try 0 launched on executor multi with executor id 3433 parsl.dataflow.dflow INFO: Standard output for task 3432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/7d806a74-b973-485a-83fe-d0a20f3afa6a_isrForCrosstalkSources_1125933_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/7d806a74-b973-485a-83fe-d0a20f3afa6a_isrForCrosstalkSources_1125933_4.stderr parsl.dataflow.dflow DEBUG: Task 3433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3433 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3433 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3433 has memoization hash 9307736bc168f3f5d074c2bea10a2d96 parsl.dataflow.memoization INFO: Task 3433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3433 try 0 launched on executor multi with executor id 3434 parsl.dataflow.dflow INFO: Standard output for task 3433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/43d5f9ce-3fff-44dc-aac5-afbc1ec39669_isrForCrosstalkSources_1125912_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/43d5f9ce-3fff-44dc-aac5-afbc1ec39669_isrForCrosstalkSources_1125912_40.stderr parsl.dataflow.dflow DEBUG: Task 3434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3434 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3434 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3434 has memoization hash 2123f7092ebed643b9034b88126ec2b3 parsl.dataflow.memoization INFO: Task 3434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3434 try 0 launched on executor multi with executor id 3435 parsl.dataflow.dflow INFO: Standard output for task 3434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/1062ed0f-81dd-4ba2-a045-f738ecf90049_isrForCrosstalkSources_1125905_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/1062ed0f-81dd-4ba2-a045-f738ecf90049_isrForCrosstalkSources_1125905_56.stderr parsl.dataflow.dflow DEBUG: Task 3435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3435 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3435 has memoization hash 61674e57c8af276465ea4546da1dcdcf parsl.dataflow.memoization INFO: Task 3435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3435 try 0 launched on executor multi with executor id 3436 parsl.dataflow.dflow INFO: Standard output for task 3435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/3724a779-46fc-4743-8369-db2785d6633a_isrForCrosstalkSources_1125929_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/3724a779-46fc-4743-8369-db2785d6633a_isrForCrosstalkSources_1125929_48.stderr parsl.dataflow.dflow DEBUG: Task 3436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3436 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3436 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3436 has memoization hash 53a7c15cdb599672f36a960e1a7e5005 parsl.dataflow.memoization INFO: Task 3436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3436 try 0 launched on executor multi with executor id 3437 parsl.dataflow.dflow INFO: Standard output for task 3436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/9bc4cb2f-1271-4efb-acc8-05aeb74ca560_isrForCrosstalkSources_1125939_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/9bc4cb2f-1271-4efb-acc8-05aeb74ca560_isrForCrosstalkSources_1125939_32.stderr parsl.dataflow.dflow DEBUG: Task 3437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3437 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3437 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3437 has memoization hash 439f3332e014db9eace265b48b652274 parsl.dataflow.memoization INFO: Task 3437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3437 try 0 launched on executor multi with executor id 3438 parsl.dataflow.dflow INFO: Standard output for task 3437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/2114426e-808d-4f40-b78d-55a866ae4355_isrForCrosstalkSources_1125917_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/2114426e-808d-4f40-b78d-55a866ae4355_isrForCrosstalkSources_1125917_36.stderr parsl.dataflow.dflow DEBUG: Task 3438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3438 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3438 has memoization hash 507bd88a8094036cfe256eddbde000fd parsl.dataflow.memoization INFO: Task 3438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3438 try 0 launched on executor multi with executor id 3439 parsl.dataflow.dflow INFO: Standard output for task 3438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/48c0fad6-8b86-4a98-bd3f-709005b6288f_isrForCrosstalkSources_1125902_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/48c0fad6-8b86-4a98-bd3f-709005b6288f_isrForCrosstalkSources_1125902_31.stderr parsl.dataflow.dflow DEBUG: Task 3439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3439 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3439 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3439 has memoization hash 812e5371d662127d2549b1597012fbb0 parsl.dataflow.memoization INFO: Task 3439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3439 try 0 launched on executor multi with executor id 3440 parsl.dataflow.dflow INFO: Standard output for task 3439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/6aed66e4-5bfa-44c7-bfd1-6dc8757ed714_isrForCrosstalkSources_1125923_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/6aed66e4-5bfa-44c7-bfd1-6dc8757ed714_isrForCrosstalkSources_1125923_2.stderr parsl.dataflow.dflow DEBUG: Task 3440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3440 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3440 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3440 has memoization hash 1f20ffa0861242b5bafc9732747dc8e6 parsl.dataflow.memoization INFO: Task 3440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3440 try 0 launched on executor multi with executor id 3441 parsl.dataflow.dflow INFO: Standard output for task 3440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/29d0a34e-3859-4516-809e-9c0de56bb4af_isrForCrosstalkSources_1125902_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/29d0a34e-3859-4516-809e-9c0de56bb4af_isrForCrosstalkSources_1125902_20.stderr parsl.dataflow.dflow DEBUG: Task 3441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3441 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3441 has memoization hash 9c623cfc793280f87e49491403453f57 parsl.dataflow.memoization INFO: Task 3441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3441 try 0 launched on executor multi with executor id 3442 parsl.dataflow.dflow INFO: Standard output for task 3441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/64d4730c-fb34-4d1c-944e-f0accda31472_isrForCrosstalkSources_1125911_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/64d4730c-fb34-4d1c-944e-f0accda31472_isrForCrosstalkSources_1125911_31.stderr parsl.dataflow.dflow DEBUG: Task 3442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3442 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3442 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3442 has memoization hash 0607fd2e0cd45f3242e2c89aa38ab318 parsl.dataflow.memoization INFO: Task 3442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3442 try 0 launched on executor multi with executor id 3443 parsl.dataflow.dflow INFO: Standard output for task 3442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/25f3a7e9-3fb4-4b71-9060-a9066aec21e1_isrForCrosstalkSources_1125939_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/25f3a7e9-3fb4-4b71-9060-a9066aec21e1_isrForCrosstalkSources_1125939_40.stderr parsl.dataflow.dflow DEBUG: Task 3443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3443 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3443 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3443 has memoization hash a6b7f9e6f570db53cd60dc53c24d7125 parsl.dataflow.memoization INFO: Task 3443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3443 try 0 launched on executor multi with executor id 3444 parsl.dataflow.dflow INFO: Standard output for task 3443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/2be33fbe-103b-430a-832e-582181f6e518_isrForCrosstalkSources_1125944_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/2be33fbe-103b-430a-832e-582181f6e518_isrForCrosstalkSources_1125944_43.stderr parsl.dataflow.dflow DEBUG: Task 3444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3444 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3444 has memoization hash 0ab964b3c3db74d0e2da75734bb6fad4 parsl.dataflow.memoization INFO: Task 3444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3444 try 0 launched on executor multi with executor id 3445 parsl.dataflow.dflow INFO: Standard output for task 3444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/101157cd-5eea-4a0d-a81f-68980040cd05_isrForCrosstalkSources_1125941_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/101157cd-5eea-4a0d-a81f-68980040cd05_isrForCrosstalkSources_1125941_21.stderr parsl.dataflow.dflow DEBUG: Task 3445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3445 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3445 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3445 has memoization hash 1623747594505e040f65453b386dbf4d parsl.dataflow.memoization INFO: Task 3445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3445 try 0 launched on executor multi with executor id 3446 parsl.dataflow.dflow INFO: Standard output for task 3445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/f977f2cc-9cdb-488d-beba-95d95e8cb8cc_isrForCrosstalkSources_1125943_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/f977f2cc-9cdb-488d-beba-95d95e8cb8cc_isrForCrosstalkSources_1125943_4.stderr parsl.dataflow.dflow DEBUG: Task 3446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3446 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3446 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3446 has memoization hash 306665c9cf4707e8ebbacdc70c41a154 parsl.dataflow.memoization INFO: Task 3446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3446 try 0 launched on executor multi with executor id 3447 parsl.dataflow.dflow INFO: Standard output for task 3446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/221f3eeb-1fcc-467b-b825-028232e06994_isrForCrosstalkSources_1125922_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/221f3eeb-1fcc-467b-b825-028232e06994_isrForCrosstalkSources_1125922_44.stderr parsl.dataflow.dflow DEBUG: Task 3447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3447 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3447 has memoization hash cc71062b824de8a5c401c4771dc48cdb parsl.dataflow.memoization INFO: Task 3447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:07 proc_lsst.multi:170 [INFO] [multi] job_id 16896444 proc_lsst.multi INFO: [multi] job_id 16896444 2024-03-07 15:07:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:07 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16896444 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16896444 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 16896444 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-07 15:07:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-07 15:07:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 15:07:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3447 try 0 launched on executor multi with executor id 3448 parsl.dataflow.dflow INFO: Standard output for task 3447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/6cfa29eb-e7bb-475d-9732-7de03e382274_isrForCrosstalkSources_1125947_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/6cfa29eb-e7bb-475d-9732-7de03e382274_isrForCrosstalkSources_1125947_10.stderr parsl.dataflow.dflow DEBUG: Task 3448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3448 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3448 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3448 has memoization hash adb248b1660e24f9db811dd864222463 parsl.dataflow.memoization INFO: Task 3448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3448 try 0 launched on executor multi with executor id 3449 parsl.dataflow.dflow INFO: Standard output for task 3448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/0b33caa4-04df-4bed-90c0-dd4de45a5e0d_isrForCrosstalkSources_1125905_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/0b33caa4-04df-4bed-90c0-dd4de45a5e0d_isrForCrosstalkSources_1125905_40.stderr parsl.dataflow.dflow DEBUG: Task 3449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3449 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3449 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3449 has memoization hash 33a4e4aade829af6426c59ba30f7725e parsl.dataflow.memoization INFO: Task 3449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3449 try 0 launched on executor multi with executor id 3450 parsl.dataflow.dflow INFO: Standard output for task 3449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/4ea113d4-5926-43f1-b31a-44a79d799ddc_isrForCrosstalkSources_1125925_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/4ea113d4-5926-43f1-b31a-44a79d799ddc_isrForCrosstalkSources_1125925_38.stderr parsl.dataflow.dflow DEBUG: Task 3450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3450 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3450 has memoization hash 4b3b9c1065deb144760bfe831b201a9b parsl.dataflow.memoization INFO: Task 3450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3450 try 0 launched on executor multi with executor id 3451 parsl.dataflow.dflow INFO: Standard output for task 3450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/676d6433-dcf0-4db2-8690-74ccacd64fd2_isrForCrosstalkSources_1125900_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/676d6433-dcf0-4db2-8690-74ccacd64fd2_isrForCrosstalkSources_1125900_57.stderr parsl.dataflow.dflow DEBUG: Task 3451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3451 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3451 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3451 has memoization hash 57e67077425a3173f0fc0a620d81523b parsl.dataflow.memoization INFO: Task 3451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3451 try 0 launched on executor multi with executor id 3452 parsl.dataflow.dflow INFO: Standard output for task 3451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/805adfd4-8de2-4e17-9839-52e046bc8ca4_isrForCrosstalkSources_1125923_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/805adfd4-8de2-4e17-9839-52e046bc8ca4_isrForCrosstalkSources_1125923_43.stderr parsl.dataflow.dflow DEBUG: Task 3452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3452 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3452 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3452 has memoization hash 8fc7be7ecf15ee0b0599ed69002d0dc9 parsl.dataflow.memoization INFO: Task 3452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39189da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3452 try 0 launched on executor multi with executor id 3453 parsl.dataflow.dflow INFO: Standard output for task 3452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/410e264d-3a19-4d68-95ac-7a1843dd122e_isrForCrosstalkSources_1125938_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/410e264d-3a19-4d68-95ac-7a1843dd122e_isrForCrosstalkSources_1125938_36.stderr parsl.dataflow.dflow DEBUG: Task 3453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3453 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3453 has memoization hash 56b28a1c12950b0d8e090b6b3f34f094 parsl.dataflow.memoization INFO: Task 3453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3453 try 0 launched on executor multi with executor id 3454 parsl.dataflow.dflow INFO: Standard output for task 3453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/028fb13a-9e51-4d92-b9d5-f3d87f23f0ef_isrForCrosstalkSources_1125902_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/028fb13a-9e51-4d92-b9d5-f3d87f23f0ef_isrForCrosstalkSources_1125902_41.stderr parsl.dataflow.dflow DEBUG: Task 3454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3454 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3454 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3454 has memoization hash 7bd8db466678f612acae07f9a7d36e63 parsl.dataflow.memoization INFO: Task 3454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3454 try 0 launched on executor multi with executor id 3455 parsl.dataflow.dflow INFO: Standard output for task 3454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/1af2a2e6-6a5e-444b-8502-04e6b00a3b27_isrForCrosstalkSources_1125936_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/1af2a2e6-6a5e-444b-8502-04e6b00a3b27_isrForCrosstalkSources_1125936_51.stderr parsl.dataflow.dflow DEBUG: Task 3455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3455 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3455 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3455 has memoization hash 95957149d59b2ff997d39af96a2e1554 parsl.dataflow.memoization INFO: Task 3455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3455 try 0 launched on executor multi with executor id 3456 parsl.dataflow.dflow INFO: Standard output for task 3455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/9e1ef4e1-23cb-4596-a798-2d82b60490b1_isrForCrosstalkSources_1125888_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/9e1ef4e1-23cb-4596-a798-2d82b60490b1_isrForCrosstalkSources_1125888_56.stderr parsl.dataflow.dflow DEBUG: Task 3456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3456 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3456 has memoization hash 175867012f2db6613045bbda645db879 parsl.dataflow.memoization INFO: Task 3456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3456 try 0 launched on executor multi with executor id 3457 parsl.dataflow.dflow INFO: Standard output for task 3456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/b341795c-594d-4213-bdfe-8f92dc040cc4_isrForCrosstalkSources_1125936_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/b341795c-594d-4213-bdfe-8f92dc040cc4_isrForCrosstalkSources_1125936_62.stderr parsl.dataflow.dflow DEBUG: Task 3457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3457 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3457 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3457 has memoization hash a83f589dcdf6ebeb282d17a99a327a33 parsl.dataflow.memoization INFO: Task 3457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3457 try 0 launched on executor multi with executor id 3458 parsl.dataflow.dflow INFO: Standard output for task 3457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/4c04d0ad-25c8-4a2e-87fa-b04a5083ac09_isrForCrosstalkSources_1125924_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/4c04d0ad-25c8-4a2e-87fa-b04a5083ac09_isrForCrosstalkSources_1125924_53.stderr parsl.dataflow.dflow DEBUG: Task 3458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3458 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3458 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3458 has memoization hash c307fe6e7bee347a793d080feef31bc1 parsl.dataflow.memoization INFO: Task 3458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3458 try 0 launched on executor multi with executor id 3459 parsl.dataflow.dflow INFO: Standard output for task 3458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/13eceab6-e4bf-4dca-b911-9794c54d8f37_isrForCrosstalkSources_1125895_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/13eceab6-e4bf-4dca-b911-9794c54d8f37_isrForCrosstalkSources_1125895_47.stderr parsl.dataflow.dflow DEBUG: Task 3459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3459 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3459 has memoization hash 645c2bc48f39bda1ece4685a58e97202 parsl.dataflow.memoization INFO: Task 3459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3459 try 0 launched on executor multi with executor id 3460 parsl.dataflow.dflow INFO: Standard output for task 3459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/38453ed6-fe65-4585-a6c0-442860505155_isrForCrosstalkSources_1125890_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/38453ed6-fe65-4585-a6c0-442860505155_isrForCrosstalkSources_1125890_23.stderr parsl.dataflow.dflow DEBUG: Task 3460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3460 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3460 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3460 has memoization hash bbe4b8a25610f345c372f9ad2a1cec4a parsl.dataflow.memoization INFO: Task 3460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3460 try 0 launched on executor multi with executor id 3461 parsl.dataflow.dflow INFO: Standard output for task 3460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/6e016326-6187-4851-88c6-90d054e4faed_isrForCrosstalkSources_1125916_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/6e016326-6187-4851-88c6-90d054e4faed_isrForCrosstalkSources_1125916_11.stderr parsl.dataflow.dflow DEBUG: Task 3461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3461 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3461 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3461 has memoization hash f15d78ec40863a46db7770fc694824c6 parsl.dataflow.memoization INFO: Task 3461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3461 try 0 launched on executor multi with executor id 3462 parsl.dataflow.dflow INFO: Standard output for task 3461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/f5a4e545-2e0b-436b-b6ed-f86a621041fa_isrForCrosstalkSources_1125928_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/f5a4e545-2e0b-436b-b6ed-f86a621041fa_isrForCrosstalkSources_1125928_29.stderr parsl.dataflow.dflow DEBUG: Task 3462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3462 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3462 has memoization hash 76ead707f87f3364bb9b93e7817345fa parsl.dataflow.memoization INFO: Task 3462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3462 try 0 launched on executor multi with executor id 3463 parsl.dataflow.dflow INFO: Standard output for task 3462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/a0f18f4b-aaab-4796-851d-b1336300e0ba_isrForCrosstalkSources_1125916_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/a0f18f4b-aaab-4796-851d-b1336300e0ba_isrForCrosstalkSources_1125916_37.stderr parsl.dataflow.dflow DEBUG: Task 3463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3463 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3463 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3463 has memoization hash ffa5d44d376f239eec8799dfddaf9a82 parsl.dataflow.memoization INFO: Task 3463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3463 try 0 launched on executor multi with executor id 3464 parsl.dataflow.dflow INFO: Standard output for task 3463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/52ebfd9b-8dc8-4e2f-9e55-d7169d212c04_isrForCrosstalkSources_1125945_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/52ebfd9b-8dc8-4e2f-9e55-d7169d212c04_isrForCrosstalkSources_1125945_49.stderr parsl.dataflow.dflow DEBUG: Task 3464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3464 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3464 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3464 has memoization hash d66f43dc069d627b897b70f6ebd0026a parsl.dataflow.memoization INFO: Task 3464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3464 try 0 launched on executor multi with executor id 3465 parsl.dataflow.dflow INFO: Standard output for task 3464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/068df873-d6a0-492e-8934-296752879429_isrForCrosstalkSources_1125935_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/068df873-d6a0-492e-8934-296752879429_isrForCrosstalkSources_1125935_57.stderr parsl.dataflow.dflow DEBUG: Task 3465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3465 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3465 has memoization hash e353a8ad991f0e38eb4bd6210b0a20c4 parsl.dataflow.memoization INFO: Task 3465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3465 try 0 launched on executor multi with executor id 3466 parsl.dataflow.dflow INFO: Standard output for task 3465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/6b8b957f-dc8a-47b4-a651-4cac02629fe0_isrForCrosstalkSources_1125906_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/6b8b957f-dc8a-47b4-a651-4cac02629fe0_isrForCrosstalkSources_1125906_27.stderr parsl.dataflow.dflow DEBUG: Task 3466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3466 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3466 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3466 has memoization hash ab88ce6cc40a35d12aaa7c0ab105c867 parsl.dataflow.memoization INFO: Task 3466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3466 try 0 launched on executor multi with executor id 3467 parsl.dataflow.dflow INFO: Standard output for task 3466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/54b4a54e-74a5-4278-a813-59ce0a5149c9_isrForCrosstalkSources_1125905_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/54b4a54e-74a5-4278-a813-59ce0a5149c9_isrForCrosstalkSources_1125905_54.stderr parsl.dataflow.dflow DEBUG: Task 3467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3467 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3467 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3467 has memoization hash 17486da978ab83fdc9b89943465ff6a8 parsl.dataflow.memoization INFO: Task 3467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3467 try 0 launched on executor multi with executor id 3468 parsl.dataflow.dflow INFO: Standard output for task 3467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/e89014a9-5190-4c1f-84b1-1e3218a6fc6e_isrForCrosstalkSources_1125911_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/e89014a9-5190-4c1f-84b1-1e3218a6fc6e_isrForCrosstalkSources_1125911_25.stderr parsl.dataflow.dflow DEBUG: Task 3468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3468 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3468 has memoization hash 7f26243ed7ab32338ffdb7de392b513e parsl.dataflow.memoization INFO: Task 3468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3468 try 0 launched on executor multi with executor id 3469 parsl.dataflow.dflow INFO: Standard output for task 3468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/cab403f1-d0c0-4242-b070-1e5071dd2b5d_isrForCrosstalkSources_1125917_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/cab403f1-d0c0-4242-b070-1e5071dd2b5d_isrForCrosstalkSources_1125917_35.stderr parsl.dataflow.dflow DEBUG: Task 3469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3469 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3469 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3469 has memoization hash 2d4f38444523ab47dbe0ea4e84e1f196 parsl.dataflow.memoization INFO: Task 3469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3469 try 0 launched on executor multi with executor id 3470 parsl.dataflow.dflow INFO: Standard output for task 3469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/3a90de7a-cf40-4860-bcf1-207176fafa8f_isrForCrosstalkSources_1125905_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/3a90de7a-cf40-4860-bcf1-207176fafa8f_isrForCrosstalkSources_1125905_62.stderr parsl.dataflow.dflow DEBUG: Task 3470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3470 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3470 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3470 has memoization hash c254fab3803d491a160a161fb6fcf898 parsl.dataflow.memoization INFO: Task 3470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3470 try 0 launched on executor multi with executor id 3471 parsl.dataflow.dflow INFO: Standard output for task 3470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/6b417d1a-0d6e-4edd-9e52-a86b2dade69c_isrForCrosstalkSources_1125945_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/6b417d1a-0d6e-4edd-9e52-a86b2dade69c_isrForCrosstalkSources_1125945_34.stderr parsl.dataflow.dflow DEBUG: Task 3471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3471 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3471 has memoization hash b8ba2fb6e22f01dd8a452b9b75a384c3 parsl.dataflow.memoization INFO: Task 3471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3471 try 0 launched on executor multi with executor id 3472 parsl.dataflow.dflow INFO: Standard output for task 3471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d6af6a82-9c47-4f52-b789-aa5917c842da_isrForCrosstalkSources_1125919_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d6af6a82-9c47-4f52-b789-aa5917c842da_isrForCrosstalkSources_1125919_37.stderr parsl.dataflow.dflow DEBUG: Task 3472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3472 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3472 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3472 has memoization hash df288e927b827ff43857d086a0bf37a6 parsl.dataflow.memoization INFO: Task 3472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:08 proc_lsst.multi:170 [INFO] [multi] job_id 16896445 proc_lsst.multi INFO: [multi] job_id 16896445 2024-03-07 15:07:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896445 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896445 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 16896445 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-07 15:07:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-07 15:07:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3472 try 0 launched on executor multi with executor id 3473 parsl.dataflow.dflow INFO: Standard output for task 3472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/4863f479-61b1-4a36-9d25-422c5366ebdd_isrForCrosstalkSources_1125946_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/4863f479-61b1-4a36-9d25-422c5366ebdd_isrForCrosstalkSources_1125946_60.stderr parsl.dataflow.dflow DEBUG: Task 3473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3473 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3473 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3473 has memoization hash c2e135911e063c763ea0be6e577a8daa parsl.dataflow.memoization INFO: Task 3473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3473 try 0 launched on executor multi with executor id 3474 parsl.dataflow.dflow INFO: Standard output for task 3473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/31ad08f6-5047-499c-bb33-b014a038b0b6_isrForCrosstalkSources_1125896_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/31ad08f6-5047-499c-bb33-b014a038b0b6_isrForCrosstalkSources_1125896_17.stderr parsl.dataflow.dflow DEBUG: Task 3474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3474 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3474 has memoization hash d2f61800e97448bc26a7b21e051b21fc parsl.dataflow.memoization INFO: Task 3474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a391887c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3474 try 0 launched on executor multi with executor id 3475 parsl.dataflow.dflow INFO: Standard output for task 3474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/090c22b1-57b6-4766-ac88-2b2c8a69e276_isrForCrosstalkSources_1125909_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/090c22b1-57b6-4766-ac88-2b2c8a69e276_isrForCrosstalkSources_1125909_19.stderr parsl.dataflow.dflow DEBUG: Task 3475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3475 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3475 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3475 has memoization hash 3a95ef7fd8fd86fb566f7641d2529f8d parsl.dataflow.memoization INFO: Task 3475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3475 try 0 launched on executor multi with executor id 3476 parsl.dataflow.dflow INFO: Standard output for task 3475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/31984ff4-7073-4254-8c5a-f92273e9aafe_isrForCrosstalkSources_1125932_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/31984ff4-7073-4254-8c5a-f92273e9aafe_isrForCrosstalkSources_1125932_15.stderr parsl.dataflow.dflow DEBUG: Task 3476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3476 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3476 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3476 has memoization hash 12a3409000daa0d55d6576cca598b67c parsl.dataflow.memoization INFO: Task 3476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3476 try 0 launched on executor multi with executor id 3477 parsl.dataflow.dflow INFO: Standard output for task 3476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/ae42c24a-e26b-420c-a1aa-bf5496ffcd22_isrForCrosstalkSources_1125888_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/ae42c24a-e26b-420c-a1aa-bf5496ffcd22_isrForCrosstalkSources_1125888_42.stderr parsl.dataflow.dflow DEBUG: Task 3477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3477 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3477 has memoization hash b24cb269ecd87d9cae192e978b768fdf parsl.dataflow.memoization INFO: Task 3477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3477 try 0 launched on executor multi with executor id 3478 parsl.dataflow.dflow INFO: Standard output for task 3477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/5060b7f1-411b-443c-b3eb-1319893a7bbd_isrForCrosstalkSources_1125892_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/5060b7f1-411b-443c-b3eb-1319893a7bbd_isrForCrosstalkSources_1125892_46.stderr parsl.dataflow.dflow DEBUG: Task 3478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3478 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3478 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3478 has memoization hash a456ccdf5c65a0f3a6bc4a5ee1ec7226 parsl.dataflow.memoization INFO: Task 3478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3478 try 0 launched on executor multi with executor id 3479 parsl.dataflow.dflow INFO: Standard output for task 3478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3491284c-d3ab-4492-9c1d-c1dd7133b916_isrForCrosstalkSources_1125941_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3491284c-d3ab-4492-9c1d-c1dd7133b916_isrForCrosstalkSources_1125941_48.stderr parsl.dataflow.dflow DEBUG: Task 3479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3479 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3479 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3479 has memoization hash 75a0441adf9619628a3997779e441008 parsl.dataflow.memoization INFO: Task 3479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3479 try 0 launched on executor multi with executor id 3480 parsl.dataflow.dflow INFO: Standard output for task 3479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/bf81715b-d484-4594-8fa0-6382693150e5_isrForCrosstalkSources_1125903_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/bf81715b-d484-4594-8fa0-6382693150e5_isrForCrosstalkSources_1125903_31.stderr parsl.dataflow.dflow DEBUG: Task 3480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3480 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3480 has memoization hash b0e43ba0709d29335924658af22f8de0 parsl.dataflow.memoization INFO: Task 3480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3480 try 0 launched on executor multi with executor id 3481 parsl.dataflow.dflow INFO: Standard output for task 3480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/7f15679d-2296-4f5c-a0c1-90993a5622b5_isrForCrosstalkSources_1125932_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/7f15679d-2296-4f5c-a0c1-90993a5622b5_isrForCrosstalkSources_1125932_11.stderr parsl.dataflow.dflow DEBUG: Task 3481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3481 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3481 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3481 has memoization hash ab0a6b130aeef8f2475476863efdb700 parsl.dataflow.memoization INFO: Task 3481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3481 try 0 launched on executor multi with executor id 3482 parsl.dataflow.dflow INFO: Standard output for task 3481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/6762db47-5f10-455e-8e7e-782aba2df2aa_isrForCrosstalkSources_1125899_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/6762db47-5f10-455e-8e7e-782aba2df2aa_isrForCrosstalkSources_1125899_55.stderr parsl.dataflow.dflow DEBUG: Task 3482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3482 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3482 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3482 has memoization hash 57887c6aed2828d0baf2d8f5e2e3b319 parsl.dataflow.memoization INFO: Task 3482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3482 try 0 launched on executor multi with executor id 3483 parsl.dataflow.dflow INFO: Standard output for task 3482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/83e1bf6c-683b-41e4-b873-b675edb3ee76_isrForCrosstalkSources_1125891_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/83e1bf6c-683b-41e4-b873-b675edb3ee76_isrForCrosstalkSources_1125891_20.stderr parsl.dataflow.dflow DEBUG: Task 3483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3483 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3483 has memoization hash 1026fe55b47264a398e4ee07a0edc99f parsl.dataflow.memoization INFO: Task 3483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3483 try 0 launched on executor multi with executor id 3484 parsl.dataflow.dflow INFO: Standard output for task 3483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/2b7d5649-5da8-4eaf-ae27-d486df580839_isrForCrosstalkSources_1125911_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/2b7d5649-5da8-4eaf-ae27-d486df580839_isrForCrosstalkSources_1125911_53.stderr parsl.dataflow.dflow DEBUG: Task 3484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3484 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3484 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3484 has memoization hash 4c0838d3d99b46f106058a25fa1f9a7e parsl.dataflow.memoization INFO: Task 3484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3484 try 0 launched on executor multi with executor id 3485 parsl.dataflow.dflow INFO: Standard output for task 3484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/81f107f4-40ec-4ac4-89bc-ac1b1b622784_isrForCrosstalkSources_1125916_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/81f107f4-40ec-4ac4-89bc-ac1b1b622784_isrForCrosstalkSources_1125916_22.stderr parsl.dataflow.dflow DEBUG: Task 3485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3485 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3485 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3485 has memoization hash efc7a083454ab0422a14c78f442973d2 parsl.dataflow.memoization INFO: Task 3485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390191c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3485 try 0 launched on executor multi with executor id 3486 parsl.dataflow.dflow INFO: Standard output for task 3485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/a4cfb9f2-f704-435e-b9c8-6aece0a642ad_isrForCrosstalkSources_1125888_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/a4cfb9f2-f704-435e-b9c8-6aece0a642ad_isrForCrosstalkSources_1125888_31.stderr parsl.dataflow.dflow DEBUG: Task 3486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3486 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3486 has memoization hash 61eb305b2a760b3a96e9d3370ec2a986 parsl.dataflow.memoization INFO: Task 3486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390194e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3486 try 0 launched on executor multi with executor id 3487 parsl.dataflow.dflow INFO: Standard output for task 3486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/eb55feea-fbc6-4b2e-9de8-ee276840041d_isrForCrosstalkSources_1125894_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/eb55feea-fbc6-4b2e-9de8-ee276840041d_isrForCrosstalkSources_1125894_45.stderr parsl.dataflow.dflow DEBUG: Task 3487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3487 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3487 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3487 has memoization hash 4197ecf28399fbe38aaee2fa847a2ece parsl.dataflow.memoization INFO: Task 3487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3487 try 0 launched on executor multi with executor id 3488 parsl.dataflow.dflow INFO: Standard output for task 3487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/97c52b5e-7e75-4b33-b666-13c6bca4640e_isrForCrosstalkSources_1125913_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/97c52b5e-7e75-4b33-b666-13c6bca4640e_isrForCrosstalkSources_1125913_45.stderr parsl.dataflow.dflow DEBUG: Task 3488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3488 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3488 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3488 has memoization hash 6d040032ebb59182615f8e7e8086f5ca parsl.dataflow.memoization INFO: Task 3488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3488 try 0 launched on executor multi with executor id 3489 parsl.dataflow.dflow INFO: Standard output for task 3488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8f023870-690f-405f-a801-e12d4ae35ab4_isrForCrosstalkSources_1125944_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8f023870-690f-405f-a801-e12d4ae35ab4_isrForCrosstalkSources_1125944_46.stderr parsl.dataflow.dflow DEBUG: Task 3489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3489 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3489 has memoization hash d413c36b64b9cc47cdd8ba02875a4fba parsl.dataflow.memoization INFO: Task 3489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3489 try 0 launched on executor multi with executor id 3490 parsl.dataflow.dflow INFO: Standard output for task 3489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/95bd77a8-1d86-4598-b01f-e51b459ef0ec_isrForCrosstalkSources_1125947_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/95bd77a8-1d86-4598-b01f-e51b459ef0ec_isrForCrosstalkSources_1125947_1.stderr parsl.dataflow.dflow DEBUG: Task 3490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3490 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3490 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3490 has memoization hash 766832efa44c30ad27824217ad5704d6 parsl.dataflow.memoization INFO: Task 3490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3490 try 0 launched on executor multi with executor id 3491 parsl.dataflow.dflow INFO: Standard output for task 3490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/7b3afb71-4f78-4830-be62-7bc959c47c34_isrForCrosstalkSources_1125937_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/7b3afb71-4f78-4830-be62-7bc959c47c34_isrForCrosstalkSources_1125937_47.stderr parsl.dataflow.dflow DEBUG: Task 3491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3491 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3491 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3491 has memoization hash ee64bb20c0dabde3800a12922bb2c131 parsl.dataflow.memoization INFO: Task 3491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3491 try 0 launched on executor multi with executor id 3492 parsl.dataflow.dflow INFO: Standard output for task 3491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/9ecd4a37-f5b3-4fe5-b880-b8a91289be84_isrForCrosstalkSources_1125929_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/9ecd4a37-f5b3-4fe5-b880-b8a91289be84_isrForCrosstalkSources_1125929_45.stderr parsl.dataflow.dflow DEBUG: Task 3492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3492 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3492 has memoization hash fc64f550f8846d1947f508ccce0a007c parsl.dataflow.memoization INFO: Task 3492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:08 proc_lsst.multi:170 [INFO] [multi] job_id 16896446 proc_lsst.multi INFO: [multi] job_id 16896446 2024-03-07 15:07:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 15:07:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896446 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896446 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 16896446 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-07 15:07:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-07 15:07:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 15:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3492 try 0 launched on executor multi with executor id 3493 parsl.dataflow.dflow INFO: Standard output for task 3492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/7d618e8d-bb51-497d-9092-55b7a40bd1ce_isrForCrosstalkSources_1125928_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/7d618e8d-bb51-497d-9092-55b7a40bd1ce_isrForCrosstalkSources_1125928_57.stderr parsl.dataflow.dflow DEBUG: Task 3493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3493 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3493 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3493 has memoization hash 301d02d5886ea67622ef97661ac69ad8 parsl.dataflow.memoization INFO: Task 3493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3493 try 0 launched on executor multi with executor id 3494 parsl.dataflow.dflow INFO: Standard output for task 3493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/b65b59cd-2cd6-4719-b8dd-e00dd4a4c4dd_isrForCrosstalkSources_1125935_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/b65b59cd-2cd6-4719-b8dd-e00dd4a4c4dd_isrForCrosstalkSources_1125935_35.stderr parsl.dataflow.dflow DEBUG: Task 3494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3494 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3494 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3494 has memoization hash 0b42891667f6f02c9df30c0ed56b0e17 parsl.dataflow.memoization INFO: Task 3494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3494 try 0 launched on executor multi with executor id 3495 parsl.dataflow.dflow INFO: Standard output for task 3494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a501d414-2836-4233-beb5-d98c911fec16_isrForCrosstalkSources_1125945_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a501d414-2836-4233-beb5-d98c911fec16_isrForCrosstalkSources_1125945_9.stderr parsl.dataflow.dflow DEBUG: Task 3495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3495 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3495 has memoization hash dabc0a2279b15f6c3c0dcac2a4768720 parsl.dataflow.memoization INFO: Task 3495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3918bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3495 try 0 launched on executor multi with executor id 3496 parsl.dataflow.dflow INFO: Standard output for task 3495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/81c99f97-532e-4992-9ce9-32fbb8c9f129_isrForCrosstalkSources_1125938_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/81c99f97-532e-4992-9ce9-32fbb8c9f129_isrForCrosstalkSources_1125938_56.stderr parsl.dataflow.dflow DEBUG: Task 3496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3496 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3496 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3496 has memoization hash c221981532bd1893a1beff17151b334a parsl.dataflow.memoization INFO: Task 3496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3496 try 0 launched on executor multi with executor id 3497 parsl.dataflow.dflow INFO: Standard output for task 3496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/a7385471-0c9d-4ea3-8f72-fb65eed4489b_isrForCrosstalkSources_1125889_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/a7385471-0c9d-4ea3-8f72-fb65eed4489b_isrForCrosstalkSources_1125889_33.stderr parsl.dataflow.dflow DEBUG: Task 3497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3497 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3497 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3497 has memoization hash 3b6c31fd3770c73cac1bb89fce8d4351 parsl.dataflow.memoization INFO: Task 3497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3497 try 0 launched on executor multi with executor id 3498 parsl.dataflow.dflow INFO: Standard output for task 3497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/838037ac-388d-4853-987d-dd8c04e43b25_isrForCrosstalkSources_1125936_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/838037ac-388d-4853-987d-dd8c04e43b25_isrForCrosstalkSources_1125936_47.stderr parsl.dataflow.dflow DEBUG: Task 3498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3498 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3498 has memoization hash 10c53f8b7f61806f00f2eda40f3aea1f parsl.dataflow.memoization INFO: Task 3498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3498 try 0 launched on executor multi with executor id 3499 parsl.dataflow.dflow INFO: Standard output for task 3498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/c66c43e1-0ec7-402e-a096-41d53264dc3c_isrForCrosstalkSources_1125900_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/c66c43e1-0ec7-402e-a096-41d53264dc3c_isrForCrosstalkSources_1125900_16.stderr parsl.dataflow.dflow DEBUG: Task 3499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3499 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3499 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3499 has memoization hash 51cfc3ff0184b0b733b044dcdd9bfa93 parsl.dataflow.memoization INFO: Task 3499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3499 try 0 launched on executor multi with executor id 3500 parsl.dataflow.dflow INFO: Standard output for task 3499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/25b7e408-e1c3-4b20-a493-573c97cac4d6_isrForCrosstalkSources_1125921_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/25b7e408-e1c3-4b20-a493-573c97cac4d6_isrForCrosstalkSources_1125921_46.stderr parsl.dataflow.dflow DEBUG: Task 3500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3500 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3500 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3500 has memoization hash af0c2ea29e75a8e673cb519ea9a30c16 parsl.dataflow.memoization INFO: Task 3500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3500 try 0 launched on executor multi with executor id 3501 parsl.dataflow.dflow INFO: Standard output for task 3500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/bcdbeaad-2745-4deb-8509-6162125f6e58_isrForCrosstalkSources_1125908_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/bcdbeaad-2745-4deb-8509-6162125f6e58_isrForCrosstalkSources_1125908_36.stderr parsl.dataflow.dflow DEBUG: Task 3501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3501 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3501 has memoization hash 382d62d9e42fc89e028c299e41332849 parsl.dataflow.memoization INFO: Task 3501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3501 try 0 launched on executor multi with executor id 3502 parsl.dataflow.dflow INFO: Standard output for task 3501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/473f7f46-6592-48ab-a851-6e21c306f878_isrForCrosstalkSources_1125916_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/473f7f46-6592-48ab-a851-6e21c306f878_isrForCrosstalkSources_1125916_20.stderr parsl.dataflow.dflow DEBUG: Task 3502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3502 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3502 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3502 has memoization hash e0d9356d071bb2dabf11c47ad42deccc parsl.dataflow.memoization INFO: Task 3502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3502 try 0 launched on executor multi with executor id 3503 parsl.dataflow.dflow INFO: Standard output for task 3502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/57b47812-143e-4c7a-94a6-c2771b459a08_isrForCrosstalkSources_1125896_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/57b47812-143e-4c7a-94a6-c2771b459a08_isrForCrosstalkSources_1125896_14.stderr parsl.dataflow.dflow DEBUG: Task 3503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3503 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3503 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3503 has memoization hash 4fe0ccdd54976dd2ae6ac7ab3f6d0b84 parsl.dataflow.memoization INFO: Task 3503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3503 try 0 launched on executor multi with executor id 3504 parsl.dataflow.dflow INFO: Standard output for task 3503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/6415c376-1d3d-462a-9529-6e43e520ac88_isrForCrosstalkSources_1125941_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/6415c376-1d3d-462a-9529-6e43e520ac88_isrForCrosstalkSources_1125941_11.stderr parsl.dataflow.dflow DEBUG: Task 3504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3504 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3504 has memoization hash 1259f0ecb557b206fb50cfc6e80c3a7d parsl.dataflow.memoization INFO: Task 3504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3504 try 0 launched on executor multi with executor id 3505 parsl.dataflow.dflow INFO: Standard output for task 3504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/1b963527-c491-4fd8-8e8b-8b7bcbfb3f2b_isrForCrosstalkSources_1125920_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/1b963527-c491-4fd8-8e8b-8b7bcbfb3f2b_isrForCrosstalkSources_1125920_6.stderr parsl.dataflow.dflow DEBUG: Task 3505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3505 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3505 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3505 has memoization hash 0b169e08216219a2b1fc9d05f80e9dc3 parsl.dataflow.memoization INFO: Task 3505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390187c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3505 try 0 launched on executor multi with executor id 3506 parsl.dataflow.dflow INFO: Standard output for task 3505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/2f2a9da4-9c23-47aa-bd26-6ad0320a2094_isrForCrosstalkSources_1125923_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/2f2a9da4-9c23-47aa-bd26-6ad0320a2094_isrForCrosstalkSources_1125923_21.stderr parsl.dataflow.dflow DEBUG: Task 3506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3506 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3506 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3506 has memoization hash f24c04e46b8b9e6e5883bd986c5d3b7f parsl.dataflow.memoization INFO: Task 3506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3506 try 0 launched on executor multi with executor id 3507 parsl.dataflow.dflow INFO: Standard output for task 3506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/b38547ba-0166-4416-b412-152c340a1d16_isrForCrosstalkSources_1125942_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/b38547ba-0166-4416-b412-152c340a1d16_isrForCrosstalkSources_1125942_54.stderr parsl.dataflow.dflow DEBUG: Task 3507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3507 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3507 has memoization hash d52e9b75a954577ac88b9a6611be7556 parsl.dataflow.memoization INFO: Task 3507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390189a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3507 try 0 launched on executor multi with executor id 3508 parsl.dataflow.dflow INFO: Standard output for task 3507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/66d34d2e-9d9e-49d4-9582-166998eb0f3b_isrForCrosstalkSources_1125933_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/66d34d2e-9d9e-49d4-9582-166998eb0f3b_isrForCrosstalkSources_1125933_62.stderr parsl.dataflow.dflow DEBUG: Task 3508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3508 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3508 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3508 has memoization hash 60e30a28bc4943d9f626e78aaadb222b parsl.dataflow.memoization INFO: Task 3508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3508 try 0 launched on executor multi with executor id 3509 parsl.dataflow.dflow INFO: Standard output for task 3508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/f19df8a7-ae56-4433-b895-3f4931f51c21_isrForCrosstalkSources_1125908_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/f19df8a7-ae56-4433-b895-3f4931f51c21_isrForCrosstalkSources_1125908_12.stderr parsl.dataflow.dflow DEBUG: Task 3509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3509 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3509 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3509 has memoization hash 7a3e4f293772675f1d5853a3c9bbcf1f parsl.dataflow.memoization INFO: Task 3509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390185e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3509 try 0 launched on executor multi with executor id 3510 parsl.dataflow.dflow INFO: Standard output for task 3509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/c457e914-6f27-40df-8c7c-7a5da6d736cf_isrForCrosstalkSources_1125943_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/c457e914-6f27-40df-8c7c-7a5da6d736cf_isrForCrosstalkSources_1125943_18.stderr parsl.dataflow.dflow DEBUG: Task 3510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3510 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3510 has memoization hash 7680c3a19962425713efb3ef40ad62da parsl.dataflow.memoization INFO: Task 3510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3510 try 0 launched on executor multi with executor id 3511 parsl.dataflow.dflow INFO: Standard output for task 3510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/ba284490-deb8-4b09-9482-ac1c4592c583_isrForCrosstalkSources_1125892_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/ba284490-deb8-4b09-9482-ac1c4592c583_isrForCrosstalkSources_1125892_34.stderr parsl.dataflow.dflow DEBUG: Task 3511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3511 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3511 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3511 has memoization hash 00c3ea9edb89c493f61d61fee62af2a0 parsl.dataflow.memoization INFO: Task 3511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3511 try 0 launched on executor multi with executor id 3512 parsl.dataflow.dflow INFO: Standard output for task 3511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/09a36002-d375-4d28-aed0-42290c7574b8_isrForCrosstalkSources_1125889_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/09a36002-d375-4d28-aed0-42290c7574b8_isrForCrosstalkSources_1125889_47.stderr parsl.dataflow.dflow DEBUG: Task 3512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3512 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3512 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3512 has memoization hash 27b05f1d0aa61d4472e80d6e1c3d1cd3 parsl.dataflow.memoization INFO: Task 3512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3512 try 0 launched on executor multi with executor id 3513 parsl.dataflow.dflow INFO: Standard output for task 3512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/09125582-0fab-4cbd-b126-113ee2ed9814_isrForCrosstalkSources_1125936_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/09125582-0fab-4cbd-b126-113ee2ed9814_isrForCrosstalkSources_1125936_50.stderr parsl.dataflow.dflow DEBUG: Task 3513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3513 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3513 has memoization hash a6a73fbfd40168aa72071d1df3ab3968 parsl.dataflow.memoization INFO: Task 3513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390184a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3513 try 0 launched on executor multi with executor id 3514 parsl.dataflow.dflow INFO: Standard output for task 3513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/ec682543-fbf7-4e42-aa87-b06edb662bb7_isrForCrosstalkSources_1125938_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/ec682543-fbf7-4e42-aa87-b06edb662bb7_isrForCrosstalkSources_1125938_10.stderr parsl.dataflow.dflow DEBUG: Task 3514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3514 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3514 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3514 has memoization hash 8ba9b418bc627f63340be63a6087f7db parsl.dataflow.memoization INFO: Task 3514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390182c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3514 try 0 launched on executor multi with executor id 3515 parsl.dataflow.dflow INFO: Standard output for task 3514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/d9deaaee-a865-48fa-9c8a-eba7fbf655fd_isrForCrosstalkSources_1125933_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/d9deaaee-a865-48fa-9c8a-eba7fbf655fd_isrForCrosstalkSources_1125933_32.stderr parsl.dataflow.dflow DEBUG: Task 3515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3515 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3515 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3515 has memoization hash c3085e5150884ea21486c117e522a063 parsl.dataflow.memoization INFO: Task 3515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3515 try 0 launched on executor multi with executor id 3516 parsl.dataflow.dflow INFO: Standard output for task 3515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/901d2002-401f-457a-b9a9-bc990e85f20e_isrForCrosstalkSources_1125939_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/901d2002-401f-457a-b9a9-bc990e85f20e_isrForCrosstalkSources_1125939_3.stderr parsl.dataflow.dflow DEBUG: Task 3516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3516 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3516 has memoization hash f2780d470e31082f4acbb53cdc049eac parsl.dataflow.memoization INFO: Task 3516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3516 try 0 launched on executor multi with executor id 3517 parsl.dataflow.dflow INFO: Standard output for task 3516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/efe1768a-b73e-4d07-bee9-907b5e3f358e_isrForCrosstalkSources_1125901_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/efe1768a-b73e-4d07-bee9-907b5e3f358e_isrForCrosstalkSources_1125901_18.stderr parsl.dataflow.dflow DEBUG: Task 3517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3517 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3517 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3517 has memoization hash a2f4f4675277649b4bf35c8d3e67522f parsl.dataflow.memoization INFO: Task 3517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3517 try 0 launched on executor multi with executor id 3518 parsl.dataflow.dflow INFO: Standard output for task 3517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/5c74197b-c6d0-43ff-abef-20bd7779bdf7_isrForCrosstalkSources_1125895_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/5c74197b-c6d0-43ff-abef-20bd7779bdf7_isrForCrosstalkSources_1125895_49.stderr parsl.dataflow.dflow DEBUG: Task 3518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3518 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3518 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3518 has memoization hash 8253fbf5081765a96a6a3ae28c39a89d parsl.dataflow.memoization INFO: Task 3518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3518 try 0 launched on executor multi with executor id 3519 parsl.dataflow.dflow INFO: Standard output for task 3518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/ffc56801-20de-4f0b-b5f5-dcc07676a56f_isrForCrosstalkSources_1125928_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/ffc56801-20de-4f0b-b5f5-dcc07676a56f_isrForCrosstalkSources_1125928_35.stderr parsl.dataflow.dflow DEBUG: Task 3519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3519 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3519 has memoization hash 2a7704e4671fda90184093a373b6f732 parsl.dataflow.memoization INFO: Task 3519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3519 try 0 launched on executor multi with executor id 3520 parsl.dataflow.dflow INFO: Standard output for task 3519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/a2f568d4-27af-4efe-b4fb-6779069745ef_isrForCrosstalkSources_1125892_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/a2f568d4-27af-4efe-b4fb-6779069745ef_isrForCrosstalkSources_1125892_12.stderr parsl.dataflow.dflow DEBUG: Task 3520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3520 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3520 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3520 has memoization hash 5bbe908812979eaf829d84ef925d1bc9 parsl.dataflow.memoization INFO: Task 3520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:08 proc_lsst.multi:170 [INFO] [multi] job_id 16896447 proc_lsst.multi INFO: [multi] job_id 16896447 2024-03-07 15:07:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 15:07:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896447 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896447 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 16896447 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-07 15:07:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-07 15:07:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 15:07:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3520 try 0 launched on executor multi with executor id 3521 parsl.dataflow.dflow INFO: Standard output for task 3520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/ef353909-3943-4451-b70a-019e4f9b7b45_isrForCrosstalkSources_1125899_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/ef353909-3943-4451-b70a-019e4f9b7b45_isrForCrosstalkSources_1125899_36.stderr parsl.dataflow.dflow DEBUG: Task 3521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3521 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3521 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3521 has memoization hash beb4ae1e083ddedf5a275d214a1272c6 parsl.dataflow.memoization INFO: Task 3521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3521 try 0 launched on executor multi with executor id 3522 parsl.dataflow.dflow INFO: Standard output for task 3521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/41815a71-1776-4fb2-86e1-3ad3abfca06a_isrForCrosstalkSources_1125944_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/41815a71-1776-4fb2-86e1-3ad3abfca06a_isrForCrosstalkSources_1125944_26.stderr parsl.dataflow.dflow DEBUG: Task 3522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3522 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3522 has memoization hash ddd3712fe9150e4090efa4450a5c305b parsl.dataflow.memoization INFO: Task 3522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3522 try 0 launched on executor multi with executor id 3523 parsl.dataflow.dflow INFO: Standard output for task 3522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/04eebbb6-cd61-4835-8db3-6cf711689c63_isrForCrosstalkSources_1125901_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/04eebbb6-cd61-4835-8db3-6cf711689c63_isrForCrosstalkSources_1125901_52.stderr parsl.dataflow.dflow DEBUG: Task 3523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3523 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3523 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3523 has memoization hash ba3d04c3fd3a6e246b96cff0ef770b16 parsl.dataflow.memoization INFO: Task 3523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3523 try 0 launched on executor multi with executor id 3524 parsl.dataflow.dflow INFO: Standard output for task 3523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/82f2dea5-fb52-46db-90c4-00aa3afce70f_isrForCrosstalkSources_1125918_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/82f2dea5-fb52-46db-90c4-00aa3afce70f_isrForCrosstalkSources_1125918_1.stderr parsl.dataflow.dflow DEBUG: Task 3524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3524 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3524 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3524 has memoization hash 9fa4e54fd39db7c7611b9199a782e02f parsl.dataflow.memoization INFO: Task 3524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3524 try 0 launched on executor multi with executor id 3525 parsl.dataflow.dflow INFO: Standard output for task 3524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/c4fff655-2d6f-4dbf-a862-e9574660cfe8_isrForCrosstalkSources_1125888_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/c4fff655-2d6f-4dbf-a862-e9574660cfe8_isrForCrosstalkSources_1125888_4.stderr parsl.dataflow.dflow DEBUG: Task 3525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3525 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3525 has memoization hash 729c37b175ec9830f92b7456f4a3bcf7 parsl.dataflow.memoization INFO: Task 3525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3525 try 0 launched on executor multi with executor id 3526 parsl.dataflow.dflow INFO: Standard output for task 3525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/76074ff3-913a-4c38-8125-10a39c6dbc37_isrForCrosstalkSources_1125889_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/76074ff3-913a-4c38-8125-10a39c6dbc37_isrForCrosstalkSources_1125889_5.stderr parsl.dataflow.dflow DEBUG: Task 3526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3526 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3526 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3526 has memoization hash 88fa018557fa039a8e26754e4d2c45f0 parsl.dataflow.memoization INFO: Task 3526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3526 try 0 launched on executor multi with executor id 3527 parsl.dataflow.dflow INFO: Standard output for task 3526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/9e998866-691e-42e9-8b69-5c634e86c6a4_isrForCrosstalkSources_1125922_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/9e998866-691e-42e9-8b69-5c634e86c6a4_isrForCrosstalkSources_1125922_12.stderr parsl.dataflow.dflow DEBUG: Task 3527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3527 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3527 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3527 has memoization hash 1d2f014890efabe8338ebdf76c3c00de parsl.dataflow.memoization INFO: Task 3527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3527 try 0 launched on executor multi with executor id 3528 parsl.dataflow.dflow INFO: Standard output for task 3527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/957f0257-fd43-441d-b610-cc4f08f71d08_isrForCrosstalkSources_1125946_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/957f0257-fd43-441d-b610-cc4f08f71d08_isrForCrosstalkSources_1125946_49.stderr parsl.dataflow.dflow DEBUG: Task 3528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3528 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3528 has memoization hash d129bb39ecc2a4008f427c8e84b9402b parsl.dataflow.memoization INFO: Task 3528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3528 try 0 launched on executor multi with executor id 3529 parsl.dataflow.dflow INFO: Standard output for task 3528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/c61caa0d-2356-435d-acda-6867d5282301_isrForCrosstalkSources_1125946_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/c61caa0d-2356-435d-acda-6867d5282301_isrForCrosstalkSources_1125946_51.stderr parsl.dataflow.dflow DEBUG: Task 3529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3529 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3529 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3529 has memoization hash acd41092dcb4f4d848d5939a8382fdb7 parsl.dataflow.memoization INFO: Task 3529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ad620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3529 try 0 launched on executor multi with executor id 3530 parsl.dataflow.dflow INFO: Standard output for task 3529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/eab24858-6666-45bf-b1d2-94a8387e6feb_isrForCrosstalkSources_1125943_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/eab24858-6666-45bf-b1d2-94a8387e6feb_isrForCrosstalkSources_1125943_47.stderr parsl.dataflow.dflow DEBUG: Task 3530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3530 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3530 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3530 has memoization hash f43afb67e04aac26a2b93527de2a3255 parsl.dataflow.memoization INFO: Task 3530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ad580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3530 try 0 launched on executor multi with executor id 3531 parsl.dataflow.dflow INFO: Standard output for task 3530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/c1509af9-97cd-4ca2-8a94-040d8a1218e4_isrForCrosstalkSources_1125918_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/c1509af9-97cd-4ca2-8a94-040d8a1218e4_isrForCrosstalkSources_1125918_42.stderr parsl.dataflow.dflow DEBUG: Task 3531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3531 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3531 has memoization hash a1ec431cd40c7c46260e72458618375b parsl.dataflow.memoization INFO: Task 3531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ad4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3531 try 0 launched on executor multi with executor id 3532 parsl.dataflow.dflow INFO: Standard output for task 3531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/75878e58-d83c-43c4-b216-86432ea07fe7_isrForCrosstalkSources_1125936_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/75878e58-d83c-43c4-b216-86432ea07fe7_isrForCrosstalkSources_1125936_20.stderr parsl.dataflow.dflow DEBUG: Task 3532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3532 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3532 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3532 has memoization hash a60b3b13458c2af097079be1673c1f84 parsl.dataflow.memoization INFO: Task 3532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ad3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3532 try 0 launched on executor multi with executor id 3533 parsl.dataflow.dflow INFO: Standard output for task 3532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/d9e75b79-f25e-4756-ad45-61a1928fe33f_isrForCrosstalkSources_1125904_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/d9e75b79-f25e-4756-ad45-61a1928fe33f_isrForCrosstalkSources_1125904_7.stderr parsl.dataflow.dflow DEBUG: Task 3533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3533 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3533 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3533 has memoization hash 85d2dfaea4e132b543c0f5ce2d552123 parsl.dataflow.memoization INFO: Task 3533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ad440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3533 try 0 launched on executor multi with executor id 3534 parsl.dataflow.dflow INFO: Standard output for task 3533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/bf92a2ec-1d71-442e-81e1-33918adc5c85_isrForCrosstalkSources_1125935_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/bf92a2ec-1d71-442e-81e1-33918adc5c85_isrForCrosstalkSources_1125935_43.stderr parsl.dataflow.dflow DEBUG: Task 3534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3534 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3534 has memoization hash 5449d278f657e67fbf294d82a9676d90 parsl.dataflow.memoization INFO: Task 3534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3534 try 0 launched on executor multi with executor id 3535 parsl.dataflow.dflow INFO: Standard output for task 3534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e847c49f-ab6e-4458-8504-a26f26495b2f_isrForCrosstalkSources_1125937_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e847c49f-ab6e-4458-8504-a26f26495b2f_isrForCrosstalkSources_1125937_46.stderr parsl.dataflow.dflow DEBUG: Task 3535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3535 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3535 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3535 has memoization hash 2bf4676b93634cdaa508e7f4b04ad114 parsl.dataflow.memoization INFO: Task 3535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390199e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3535 try 0 launched on executor multi with executor id 3536 parsl.dataflow.dflow INFO: Standard output for task 3535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/709c70f4-2a25-45d8-b95d-5b60bc24cf4f_isrForCrosstalkSources_1125917_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/709c70f4-2a25-45d8-b95d-5b60bc24cf4f_isrForCrosstalkSources_1125917_14.stderr parsl.dataflow.dflow DEBUG: Task 3536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3536 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3536 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3536 has memoization hash b00ad6baf4da22f75cf9717bf6610be4 parsl.dataflow.memoization INFO: Task 3536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3536 try 0 launched on executor multi with executor id 3537 parsl.dataflow.dflow INFO: Standard output for task 3536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/59b232e9-4d1b-4036-8da7-64741ff03c66_isrForCrosstalkSources_1125934_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/59b232e9-4d1b-4036-8da7-64741ff03c66_isrForCrosstalkSources_1125934_57.stderr parsl.dataflow.dflow DEBUG: Task 3537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3537 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3537 has memoization hash da1fdbe9c300dd83a49a23e0a513f785 parsl.dataflow.memoization INFO: Task 3537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3537 try 0 launched on executor multi with executor id 3538 parsl.dataflow.dflow INFO: Standard output for task 3537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/f815da8a-82f2-4f9e-beee-3c921b30c0ff_isrForCrosstalkSources_1125942_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/f815da8a-82f2-4f9e-beee-3c921b30c0ff_isrForCrosstalkSources_1125942_38.stderr parsl.dataflow.dflow DEBUG: Task 3538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3538 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3538 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3538 has memoization hash 42e2a6074f8e3670e14efd26222a8ddf parsl.dataflow.memoization INFO: Task 3538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3538 try 0 launched on executor multi with executor id 3539 parsl.dataflow.dflow INFO: Standard output for task 3538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/948a95c6-ee36-4acd-8152-ce6c774427a5_isrForCrosstalkSources_1125935_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/948a95c6-ee36-4acd-8152-ce6c774427a5_isrForCrosstalkSources_1125935_54.stderr parsl.dataflow.dflow DEBUG: Task 3539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3539 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3539 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3539 has memoization hash feb2817069151a8cacd89893aeb0a152 parsl.dataflow.memoization INFO: Task 3539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3539 try 0 launched on executor multi with executor id 3540 parsl.dataflow.dflow INFO: Standard output for task 3539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/601509f2-dbcd-4aec-9941-29573b96e6bb_isrForCrosstalkSources_1125928_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/601509f2-dbcd-4aec-9941-29573b96e6bb_isrForCrosstalkSources_1125928_4.stderr parsl.dataflow.dflow DEBUG: Task 3540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3540 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3540 has memoization hash 9b384dd5d701ae2ffe7cbde1f22e26d7 parsl.dataflow.memoization INFO: Task 3540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3540 try 0 launched on executor multi with executor id 3541 parsl.dataflow.dflow INFO: Standard output for task 3540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/f2a529c5-410c-409b-91e1-f0ab5dc2191f_isrForCrosstalkSources_1125911_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/f2a529c5-410c-409b-91e1-f0ab5dc2191f_isrForCrosstalkSources_1125911_23.stderr parsl.dataflow.dflow DEBUG: Task 3541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3541 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3541 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3541 has memoization hash f6e0f4879ec72b73b34f255949be0321 parsl.dataflow.memoization INFO: Task 3541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3541 try 0 launched on executor multi with executor id 3542 parsl.dataflow.dflow INFO: Standard output for task 3541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/e8118866-cd33-4945-a6e9-6515b95333d4_isrForCrosstalkSources_1125923_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/e8118866-cd33-4945-a6e9-6515b95333d4_isrForCrosstalkSources_1125923_3.stderr parsl.dataflow.dflow DEBUG: Task 3542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3542 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3542 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3542 has memoization hash 2cb50d799b62ecd9ca74418ed8665489 parsl.dataflow.memoization INFO: Task 3542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3542 try 0 launched on executor multi with executor id 3543 parsl.dataflow.dflow INFO: Standard output for task 3542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/351a2108-a1ca-42ce-bba9-b936531c47b8_isrForCrosstalkSources_1125902_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/351a2108-a1ca-42ce-bba9-b936531c47b8_isrForCrosstalkSources_1125902_9.stderr parsl.dataflow.dflow DEBUG: Task 3543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3543 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3543 has memoization hash 3ddc89515ab64389af48dc06354c617c parsl.dataflow.memoization INFO: Task 3543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3543 try 0 launched on executor multi with executor id 3544 parsl.dataflow.dflow INFO: Standard output for task 3543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/88e9fae8-dfd0-4a18-bfa2-60e8d0ed0d55_isrForCrosstalkSources_1125944_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/88e9fae8-dfd0-4a18-bfa2-60e8d0ed0d55_isrForCrosstalkSources_1125944_4.stderr parsl.dataflow.dflow DEBUG: Task 3544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3544 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3544 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3544 has memoization hash 60acb94872ba42c5c1a7accaa7aba98c parsl.dataflow.memoization INFO: Task 3544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3544 try 0 launched on executor multi with executor id 3545 parsl.dataflow.dflow INFO: Standard output for task 3544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d82445eb-dadc-4b0d-98c0-37012293459b_isrForCrosstalkSources_1125915_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d82445eb-dadc-4b0d-98c0-37012293459b_isrForCrosstalkSources_1125915_11.stderr parsl.dataflow.dflow DEBUG: Task 3545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3545 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3545 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3545 has memoization hash a89c18365ea59fd2d0dc57432e6b00e6 parsl.dataflow.memoization INFO: Task 3545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3545 try 0 launched on executor multi with executor id 3546 parsl.dataflow.dflow INFO: Standard output for task 3545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/0047afed-ceed-4dad-93c7-5d4b5dfd70d3_isrForCrosstalkSources_1125916_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/0047afed-ceed-4dad-93c7-5d4b5dfd70d3_isrForCrosstalkSources_1125916_48.stderr parsl.dataflow.dflow DEBUG: Task 3546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3546 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3546 has memoization hash de7b066e1d0728515387a83a7b15b603 parsl.dataflow.memoization INFO: Task 3546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3546 try 0 launched on executor multi with executor id 3547 parsl.dataflow.dflow INFO: Standard output for task 3546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/8a6b8317-cc89-445a-b419-9555e9b6db4a_isrForCrosstalkSources_1125909_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/8a6b8317-cc89-445a-b419-9555e9b6db4a_isrForCrosstalkSources_1125909_17.stderr parsl.dataflow.dflow DEBUG: Task 3547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3547 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3547 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3547 has memoization hash 361fea3f40dba9f925765f894759f7e7 parsl.dataflow.memoization INFO: Task 3547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3547 try 0 launched on executor multi with executor id 3548 parsl.dataflow.dflow INFO: Standard output for task 3547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/303a5ef6-ea65-4b04-b504-83b40b866613_isrForCrosstalkSources_1125896_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/303a5ef6-ea65-4b04-b504-83b40b866613_isrForCrosstalkSources_1125896_13.stderr parsl.dataflow.dflow DEBUG: Task 3548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3548 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3548 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3548 has memoization hash 37e2f617c58667baf1e7e44a4aa79b87 parsl.dataflow.memoization INFO: Task 3548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3548 try 0 launched on executor multi with executor id 3549 parsl.dataflow.dflow INFO: Standard output for task 3548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/6d31d861-bcbc-48bc-a087-d925afd4d847_isrForCrosstalkSources_1125925_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/6d31d861-bcbc-48bc-a087-d925afd4d847_isrForCrosstalkSources_1125925_27.stderr parsl.dataflow.dflow DEBUG: Task 3549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3549 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3549 has memoization hash db56faa27e936ab312bbd9ba8b24bacf parsl.dataflow.memoization INFO: Task 3549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3549 try 0 launched on executor multi with executor id 3550 parsl.dataflow.dflow INFO: Standard output for task 3549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/e0427cd6-5316-40eb-8c3a-1892d64ff6b3_isrForCrosstalkSources_1125935_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/e0427cd6-5316-40eb-8c3a-1892d64ff6b3_isrForCrosstalkSources_1125935_44.stderr parsl.dataflow.dflow DEBUG: Task 3550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3550 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3550 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3550 has memoization hash e0409fd5615118bdcb598bd6252cc004 parsl.dataflow.memoization INFO: Task 3550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3550 try 0 launched on executor multi with executor id 3551 parsl.dataflow.dflow INFO: Standard output for task 3550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/3647c19f-7ee6-492e-bf7e-dd7a7c4df4c6_isrForCrosstalkSources_1125904_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/3647c19f-7ee6-492e-bf7e-dd7a7c4df4c6_isrForCrosstalkSources_1125904_3.stderr parsl.dataflow.dflow DEBUG: Task 3551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3551 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3551 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3551 has memoization hash ef9f871bf3e9b610b3926896614d0a4d parsl.dataflow.memoization INFO: Task 3551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3551 try 0 launched on executor multi with executor id 3552 parsl.dataflow.dflow INFO: Standard output for task 3551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/07a943b8-9b21-441d-a392-c604f4ac1d5c_isrForCrosstalkSources_1125940_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/07a943b8-9b21-441d-a392-c604f4ac1d5c_isrForCrosstalkSources_1125940_55.stderr parsl.dataflow.dflow DEBUG: Task 3552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3552 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3552 has memoization hash 6bf119df5fea8a779cdd2145e4e12158 parsl.dataflow.memoization INFO: Task 3552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3552 try 0 launched on executor multi with executor id 3553 parsl.dataflow.dflow INFO: Standard output for task 3552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/b24537e5-0d92-44d4-aead-25298829177c_isrForCrosstalkSources_1125895_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/b24537e5-0d92-44d4-aead-25298829177c_isrForCrosstalkSources_1125895_11.stderr parsl.dataflow.dflow DEBUG: Task 3553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3553 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3553 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3553 has memoization hash ab8ecd725b37ca20ca5fff7e670faa10 parsl.dataflow.memoization INFO: Task 3553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3553 try 0 launched on executor multi with executor id 3554 parsl.dataflow.dflow INFO: Standard output for task 3553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f621b493-ead9-43af-9cd4-dc681002a098_isrForCrosstalkSources_1125910_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f621b493-ead9-43af-9cd4-dc681002a098_isrForCrosstalkSources_1125910_11.stderr parsl.dataflow.dflow DEBUG: Task 3554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3554 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3554 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3554 has memoization hash a0d3773c31e82374ec4bf7e187586b5b parsl.dataflow.memoization INFO: Task 3554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:09 proc_lsst.multi:170 [INFO] [multi] job_id 16896448 proc_lsst.multi INFO: [multi] job_id 16896448 2024-03-07 15:07:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 15:07:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896448 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896448 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 16896448 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-07 15:07:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-07 15:07:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 15:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3554 try 0 launched on executor multi with executor id 3555 parsl.dataflow.dflow INFO: Standard output for task 3554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/9b1213da-7d8c-4110-bd88-571c0c56f5ab_isrForCrosstalkSources_1125891_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/9b1213da-7d8c-4110-bd88-571c0c56f5ab_isrForCrosstalkSources_1125891_38.stderr parsl.dataflow.dflow DEBUG: Task 3555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3555 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3555 has memoization hash 3dfa73b638402d94ef8a683c0336e237 parsl.dataflow.memoization INFO: Task 3555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3555 try 0 launched on executor multi with executor id 3556 parsl.dataflow.dflow INFO: Standard output for task 3555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/d13f2e06-8d57-4dd2-85cb-d42700144f91_isrForCrosstalkSources_1125922_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/d13f2e06-8d57-4dd2-85cb-d42700144f91_isrForCrosstalkSources_1125922_9.stderr parsl.dataflow.dflow DEBUG: Task 3556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3556 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3556 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3556 has memoization hash 70f7bf2e294e78af9241e73ea37f8ce7 parsl.dataflow.memoization INFO: Task 3556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3556 try 0 launched on executor multi with executor id 3557 parsl.dataflow.dflow INFO: Standard output for task 3556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/f37f8cfd-f38c-4aef-8bc6-3617b9f166a1_isrForCrosstalkSources_1125899_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/f37f8cfd-f38c-4aef-8bc6-3617b9f166a1_isrForCrosstalkSources_1125899_59.stderr parsl.dataflow.dflow DEBUG: Task 3557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3557 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3557 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3557 has memoization hash 6b398ad817b7b2933c7508cd016bc105 parsl.dataflow.memoization INFO: Task 3557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3557 try 0 launched on executor multi with executor id 3558 parsl.dataflow.dflow INFO: Standard output for task 3557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/f30ab3e5-6bf4-4bed-acd6-d7f5f49e7eb2_isrForCrosstalkSources_1125934_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/f30ab3e5-6bf4-4bed-acd6-d7f5f49e7eb2_isrForCrosstalkSources_1125934_22.stderr parsl.dataflow.dflow DEBUG: Task 3558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3558 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3558 has memoization hash 53a2df01a782d092e28e296f0ab02fd5 parsl.dataflow.memoization INFO: Task 3558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3558 try 0 launched on executor multi with executor id 3559 parsl.dataflow.dflow INFO: Standard output for task 3558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/5e6b9290-7ef5-4a84-8116-bf3b9279aa21_isrForCrosstalkSources_1125932_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/5e6b9290-7ef5-4a84-8116-bf3b9279aa21_isrForCrosstalkSources_1125932_14.stderr parsl.dataflow.dflow DEBUG: Task 3559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3559 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3559 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3559 has memoization hash cad454c8fb9a78e238b48de25eea10eb parsl.dataflow.memoization INFO: Task 3559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3559 try 0 launched on executor multi with executor id 3560 parsl.dataflow.dflow INFO: Standard output for task 3559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/d65fba4a-4b82-41f5-85a4-399e19640e7d_isrForCrosstalkSources_1125896_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/d65fba4a-4b82-41f5-85a4-399e19640e7d_isrForCrosstalkSources_1125896_24.stderr parsl.dataflow.dflow DEBUG: Task 3560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3560 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3560 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3560 has memoization hash ed827a980cb5e7813c04be2ae2692308 parsl.dataflow.memoization INFO: Task 3560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390aea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3560 try 0 launched on executor multi with executor id 3561 parsl.dataflow.dflow INFO: Standard output for task 3560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/d18ea9e4-c86a-4aeb-bd1d-07d43395ce44_isrForCrosstalkSources_1125938_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/d18ea9e4-c86a-4aeb-bd1d-07d43395ce44_isrForCrosstalkSources_1125938_43.stderr parsl.dataflow.dflow DEBUG: Task 3561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3561 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3561 has memoization hash 2655a8b2ec6e91a49921718754b221fb parsl.dataflow.memoization INFO: Task 3561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ae8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3561 try 0 launched on executor multi with executor id 3562 parsl.dataflow.dflow INFO: Standard output for task 3561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/06e8ac17-ff06-4a57-aa18-c017fe5caba2_isrForCrosstalkSources_1125897_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/06e8ac17-ff06-4a57-aa18-c017fe5caba2_isrForCrosstalkSources_1125897_32.stderr parsl.dataflow.dflow DEBUG: Task 3562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3562 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3562 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3562 has memoization hash 4eac50d2f230f90dd4171308164d5ccf parsl.dataflow.memoization INFO: Task 3562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ae7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3562 try 0 launched on executor multi with executor id 3563 parsl.dataflow.dflow INFO: Standard output for task 3562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9c0ead5b-6e52-4cf6-a3a3-6169bb591688_isrForCrosstalkSources_1125934_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9c0ead5b-6e52-4cf6-a3a3-6169bb591688_isrForCrosstalkSources_1125934_40.stderr parsl.dataflow.dflow DEBUG: Task 3563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3563 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3563 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3563 has memoization hash aa9d83e9c120c61f66f30959e611a2a1 parsl.dataflow.memoization INFO: Task 3563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ae840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3563 try 0 launched on executor multi with executor id 3564 parsl.dataflow.dflow INFO: Standard output for task 3563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/2659ae4c-adc9-4161-9f24-a3391e511350_isrForCrosstalkSources_1125940_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/2659ae4c-adc9-4161-9f24-a3391e511350_isrForCrosstalkSources_1125940_37.stderr parsl.dataflow.dflow DEBUG: Task 3564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3564 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3564 has memoization hash c0213924d415c96148dc95374fc45ec6 parsl.dataflow.memoization INFO: Task 3564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ae700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3564 try 0 launched on executor multi with executor id 3565 parsl.dataflow.dflow INFO: Standard output for task 3564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/5cf16b00-6b36-4b09-bac4-313d2569e601_isrForCrosstalkSources_1125933_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/5cf16b00-6b36-4b09-bac4-313d2569e601_isrForCrosstalkSources_1125933_15.stderr parsl.dataflow.dflow DEBUG: Task 3565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3565 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3565 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3565 has memoization hash 2a6113716da55b19ad651063d075fdfe parsl.dataflow.memoization INFO: Task 3565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ae660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3565 try 0 launched on executor multi with executor id 3566 parsl.dataflow.dflow INFO: Standard output for task 3565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/385b5a8f-ae5b-42f7-aedc-734681e7e69f_isrForCrosstalkSources_1125918_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/385b5a8f-ae5b-42f7-aedc-734681e7e69f_isrForCrosstalkSources_1125918_25.stderr parsl.dataflow.dflow DEBUG: Task 3566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3566 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3566 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3566 has memoization hash ecd1c1feba6462c2a4776a860e49a672 parsl.dataflow.memoization INFO: Task 3566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ae5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3566 try 0 launched on executor multi with executor id 3567 parsl.dataflow.dflow INFO: Standard output for task 3566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/fbe01cff-bb00-4265-9ec0-69a17d304b03_isrForCrosstalkSources_1125892_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/fbe01cff-bb00-4265-9ec0-69a17d304b03_isrForCrosstalkSources_1125892_62.stderr parsl.dataflow.dflow DEBUG: Task 3567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3567 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3567 has memoization hash ba22f9b832001e1a78a631c5a73c7f7c parsl.dataflow.memoization INFO: Task 3567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3567 try 0 launched on executor multi with executor id 3568 parsl.dataflow.dflow INFO: Standard output for task 3567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/7b6e9d98-5833-473d-9856-9ed0606e6466_isrForCrosstalkSources_1125894_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/7b6e9d98-5833-473d-9856-9ed0606e6466_isrForCrosstalkSources_1125894_29.stderr parsl.dataflow.dflow DEBUG: Task 3568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3568 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3568 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3568 has memoization hash 67388f01a2e95cfe0b143ff035bfd474 parsl.dataflow.memoization INFO: Task 3568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3568 try 0 launched on executor multi with executor id 3569 parsl.dataflow.dflow INFO: Standard output for task 3568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/3024035f-bac4-413e-864c-d7c05468b472_isrForCrosstalkSources_1125891_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/3024035f-bac4-413e-864c-d7c05468b472_isrForCrosstalkSources_1125891_17.stderr parsl.dataflow.dflow DEBUG: Task 3569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3569 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3569 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3569 has memoization hash 8b9dcefe6195e0f4bf856aa3a0dd559b parsl.dataflow.memoization INFO: Task 3569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3569 try 0 launched on executor multi with executor id 3570 parsl.dataflow.dflow INFO: Standard output for task 3569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/4137e972-e04a-4893-9294-340ce49579d3_isrForCrosstalkSources_1125916_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/4137e972-e04a-4893-9294-340ce49579d3_isrForCrosstalkSources_1125916_56.stderr parsl.dataflow.dflow DEBUG: Task 3570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3570 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3570 has memoization hash 83d8ef8ca161ea4d6c3886b380036c73 parsl.dataflow.memoization INFO: Task 3570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3570 try 0 launched on executor multi with executor id 3571 parsl.dataflow.dflow INFO: Standard output for task 3570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/146d0270-b61e-4857-83e4-0074b88d7bbe_isrForCrosstalkSources_1125927_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/146d0270-b61e-4857-83e4-0074b88d7bbe_isrForCrosstalkSources_1125927_38.stderr parsl.dataflow.dflow DEBUG: Task 3571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3571 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3571 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3571 has memoization hash acc20d93b7a9336ebb67ef3a8bb3f89b parsl.dataflow.memoization INFO: Task 3571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3571 try 0 launched on executor multi with executor id 3572 parsl.dataflow.dflow INFO: Standard output for task 3571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/21b8b7ff-8605-44ba-b115-a7235d28d8a4_isrForCrosstalkSources_1125929_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/21b8b7ff-8605-44ba-b115-a7235d28d8a4_isrForCrosstalkSources_1125929_16.stderr parsl.dataflow.dflow DEBUG: Task 3572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3572 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3572 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3572 has memoization hash e2661289af354900a891ebc615dc55b2 parsl.dataflow.memoization INFO: Task 3572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3572 try 0 launched on executor multi with executor id 3573 parsl.dataflow.dflow INFO: Standard output for task 3572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/e05445d0-c7fa-4423-adcc-8690a4b41d2e_isrForCrosstalkSources_1125928_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/e05445d0-c7fa-4423-adcc-8690a4b41d2e_isrForCrosstalkSources_1125928_22.stderr parsl.dataflow.dflow DEBUG: Task 3573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3573 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3573 has memoization hash bcb6d809571b495add26815978c2542e parsl.dataflow.memoization INFO: Task 3573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3573 try 0 launched on executor multi with executor id 3574 parsl.dataflow.dflow INFO: Standard output for task 3573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/d9c09e9d-4275-4018-a0cf-503a67903d90_isrForCrosstalkSources_1125906_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/d9c09e9d-4275-4018-a0cf-503a67903d90_isrForCrosstalkSources_1125906_3.stderr parsl.dataflow.dflow DEBUG: Task 3574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3574 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3574 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3574 has memoization hash 1706da7b37229b5cd5d2036cdb97441f parsl.dataflow.memoization INFO: Task 3574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3574 try 0 launched on executor multi with executor id 3575 parsl.dataflow.dflow INFO: Standard output for task 3574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/69b3af32-fce1-4600-a338-96abaa8cae6d_isrForCrosstalkSources_1125922_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/69b3af32-fce1-4600-a338-96abaa8cae6d_isrForCrosstalkSources_1125922_59.stderr parsl.dataflow.dflow DEBUG: Task 3575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3575 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3575 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3575 has memoization hash ca7bd32bb35bb01c78e04426fb345d65 parsl.dataflow.memoization INFO: Task 3575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3575 try 0 launched on executor multi with executor id 3576 parsl.dataflow.dflow INFO: Standard output for task 3575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c5ed509d-d7b2-4d53-8475-820ecb0a33c1_isrForCrosstalkSources_1125917_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c5ed509d-d7b2-4d53-8475-820ecb0a33c1_isrForCrosstalkSources_1125917_32.stderr parsl.dataflow.dflow DEBUG: Task 3576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3576 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3576 has memoization hash b2b323e2f65b4ee1cf2291db555c287a parsl.dataflow.memoization INFO: Task 3576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3576 try 0 launched on executor multi with executor id 3577 parsl.dataflow.dflow INFO: Standard output for task 3576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/925326ae-7c3a-427c-984b-7f3e54498a88_isrForCrosstalkSources_1125913_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/925326ae-7c3a-427c-984b-7f3e54498a88_isrForCrosstalkSources_1125913_48.stderr parsl.dataflow.dflow DEBUG: Task 3577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3577 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3577 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3577 has memoization hash f9cdf6379f5b9b951f40f045fe9c8bfb parsl.dataflow.memoization INFO: Task 3577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390198a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3577 try 0 launched on executor multi with executor id 3578 parsl.dataflow.dflow INFO: Standard output for task 3577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/50d0b82d-66a0-427f-89f4-8140a4286a86_isrForCrosstalkSources_1125902_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/50d0b82d-66a0-427f-89f4-8140a4286a86_isrForCrosstalkSources_1125902_58.stderr parsl.dataflow.dflow DEBUG: Task 3578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3578 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3578 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3578 has memoization hash 7d4721555b74b8192849d4e5f91063e3 parsl.dataflow.memoization INFO: Task 3578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3578 try 0 launched on executor multi with executor id 3579 parsl.dataflow.dflow INFO: Standard output for task 3578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/38f7f124-6c42-423a-ac59-e825e7ce6cb7_isrForCrosstalkSources_1125898_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/38f7f124-6c42-423a-ac59-e825e7ce6cb7_isrForCrosstalkSources_1125898_34.stderr parsl.dataflow.dflow DEBUG: Task 3579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3579 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3579 has memoization hash 70ae99036c8b235e658fa01f63f9cbcb parsl.dataflow.memoization INFO: Task 3579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390196c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3579 try 0 launched on executor multi with executor id 3580 parsl.dataflow.dflow INFO: Standard output for task 3579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/2c8b40ce-e25e-4e10-aa20-ccb621806e0b_isrForCrosstalkSources_1125897_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/2c8b40ce-e25e-4e10-aa20-ccb621806e0b_isrForCrosstalkSources_1125897_15.stderr parsl.dataflow.dflow DEBUG: Task 3580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3580 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3580 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3580 has memoization hash d17fd6f928b0a5e33f3af5009ece26fa parsl.dataflow.memoization INFO: Task 3580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390180e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3580 try 0 launched on executor multi with executor id 3581 parsl.dataflow.dflow INFO: Standard output for task 3580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d094d585-ae52-4110-96ed-2df934b315e9_isrForCrosstalkSources_1125915_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d094d585-ae52-4110-96ed-2df934b315e9_isrForCrosstalkSources_1125915_2.stderr parsl.dataflow.dflow DEBUG: Task 3581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3581 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3581 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3581 has memoization hash 4eb75f78008682ef67633540af1c1b7d parsl.dataflow.memoization INFO: Task 3581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3581 try 0 launched on executor multi with executor id 3582 parsl.dataflow.dflow INFO: Standard output for task 3581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/ab584911-65ae-4997-94d2-776c20b2b1bb_isrForCrosstalkSources_1125890_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/ab584911-65ae-4997-94d2-776c20b2b1bb_isrForCrosstalkSources_1125890_8.stderr parsl.dataflow.dflow DEBUG: Task 3582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3582 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3582 has memoization hash f32ee143ef8627863b10c4061ccf8a1e parsl.dataflow.memoization INFO: Task 3582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3582 try 0 launched on executor multi with executor id 3583 parsl.dataflow.dflow INFO: Standard output for task 3582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/6b1f4d88-4aca-4c7b-bae9-b9e61f5e1aac_isrForCrosstalkSources_1125930_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/6b1f4d88-4aca-4c7b-bae9-b9e61f5e1aac_isrForCrosstalkSources_1125930_52.stderr parsl.dataflow.dflow DEBUG: Task 3583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3583 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3583 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3583 has memoization hash 89e6b54681c10d17d1fff0e7b8cd78fd parsl.dataflow.memoization INFO: Task 3583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3583 try 0 launched on executor multi with executor id 3584 parsl.dataflow.dflow INFO: Standard output for task 3583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/9e3a368f-405f-494a-bf8b-f5627703b427_isrForCrosstalkSources_1125896_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/9e3a368f-405f-494a-bf8b-f5627703b427_isrForCrosstalkSources_1125896_41.stderr parsl.dataflow.dflow DEBUG: Task 3584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3584 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3584 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3584 has memoization hash ebd82663ec915f65f9d1fd58c1af4630 parsl.dataflow.memoization INFO: Task 3584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:09 proc_lsst.multi:170 [INFO] [multi] job_id 16896449 proc_lsst.multi INFO: [multi] job_id 16896449 2024-03-07 15:07:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896449 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896449 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 16896449 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-07 15:07:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-07 15:07:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3584 try 0 launched on executor multi with executor id 3585 parsl.dataflow.dflow INFO: Standard output for task 3584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/164897fb-ac12-4d77-9193-945d0d9ff2e7_isrForCrosstalkSources_1125937_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/164897fb-ac12-4d77-9193-945d0d9ff2e7_isrForCrosstalkSources_1125937_62.stderr parsl.dataflow.dflow DEBUG: Task 3585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3585 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3585 has memoization hash 3a90eb4ce75b2791fcb3baf054c01edb parsl.dataflow.memoization INFO: Task 3585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39018360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3585 try 0 launched on executor multi with executor id 3586 parsl.dataflow.dflow INFO: Standard output for task 3585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/0d50e03b-65a4-4b72-874c-b7d26d326316_isrForCrosstalkSources_1125900_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/0d50e03b-65a4-4b72-874c-b7d26d326316_isrForCrosstalkSources_1125900_39.stderr parsl.dataflow.dflow DEBUG: Task 3586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3586 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3586 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3586 has memoization hash 99f41c7aa0666fb46619ea283526bb5b parsl.dataflow.memoization INFO: Task 3586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3586 try 0 launched on executor multi with executor id 3587 parsl.dataflow.dflow INFO: Standard output for task 3586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/b906ca29-4406-4b19-b627-49883510e5b3_isrForCrosstalkSources_1125943_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/b906ca29-4406-4b19-b627-49883510e5b3_isrForCrosstalkSources_1125943_2.stderr parsl.dataflow.dflow DEBUG: Task 3587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3587 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3587 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3587 has memoization hash 94d806518cc849d1fd113523b3682077 parsl.dataflow.memoization INFO: Task 3587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a39019d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3587 try 0 launched on executor multi with executor id 3588 parsl.dataflow.dflow INFO: Standard output for task 3587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/b94b8344-6eb5-40e4-862a-9aa04207d75d_isrForCrosstalkSources_1125892_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/b94b8344-6eb5-40e4-862a-9aa04207d75d_isrForCrosstalkSources_1125892_9.stderr parsl.dataflow.dflow DEBUG: Task 3588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3588 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3588 has memoization hash a8bef1b7f978e8aa79060c8be062c166 parsl.dataflow.memoization INFO: Task 3588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3588 try 0 launched on executor multi with executor id 3589 parsl.dataflow.dflow INFO: Standard output for task 3588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/98d0a7e7-5604-4438-a3a7-ef0226e9dcc5_isrForCrosstalkSources_1125923_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/98d0a7e7-5604-4438-a3a7-ef0226e9dcc5_isrForCrosstalkSources_1125923_33.stderr parsl.dataflow.dflow DEBUG: Task 3589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3589 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3589 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3589 has memoization hash 48e8ab548939ddc027d653ff606e82f5 parsl.dataflow.memoization INFO: Task 3589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a3901bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3589 try 0 launched on executor multi with executor id 3590 parsl.dataflow.dflow INFO: Standard output for task 3589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/5111c325-bdcb-4a55-a285-972ae3b10bee_isrForCrosstalkSources_1125894_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/5111c325-bdcb-4a55-a285-972ae3b10bee_isrForCrosstalkSources_1125894_15.stderr parsl.dataflow.dflow DEBUG: Task 3590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3590 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3590 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3590 has memoization hash 210818d45a653b0d132a77e4c249626f parsl.dataflow.memoization INFO: Task 3590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390afc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3590 try 0 launched on executor multi with executor id 3591 parsl.dataflow.dflow INFO: Standard output for task 3590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/ba49e149-8831-46fe-9f72-b79fdd940697_isrForCrosstalkSources_1125924_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/ba49e149-8831-46fe-9f72-b79fdd940697_isrForCrosstalkSources_1125924_8.stderr parsl.dataflow.dflow DEBUG: Task 3591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3591 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3591 has memoization hash 21d1dd613c0962b56c987f1357d2c152 parsl.dataflow.memoization INFO: Task 3591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390afba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3591 try 0 launched on executor multi with executor id 3592 parsl.dataflow.dflow INFO: Standard output for task 3591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/86e9b3db-e586-4111-b87d-ff75d563960e_isrForCrosstalkSources_1125937_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/86e9b3db-e586-4111-b87d-ff75d563960e_isrForCrosstalkSources_1125937_34.stderr parsl.dataflow.dflow DEBUG: Task 3592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3592 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3592 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3592 has memoization hash 9005e8b8f9f91e09fbdd49db9de528f4 parsl.dataflow.memoization INFO: Task 3592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390afa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3592 try 0 launched on executor multi with executor id 3593 parsl.dataflow.dflow INFO: Standard output for task 3592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6c83a70d-d4e5-4756-932b-c706be95a3ef_isrForCrosstalkSources_1125909_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6c83a70d-d4e5-4756-932b-c706be95a3ef_isrForCrosstalkSources_1125909_56.stderr parsl.dataflow.dflow DEBUG: Task 3593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3593 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3593 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3593 has memoization hash 5b32b439e5c8dfc55f0b76f95b308304 parsl.dataflow.memoization INFO: Task 3593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390afb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3593 try 0 launched on executor multi with executor id 3594 parsl.dataflow.dflow INFO: Standard output for task 3593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/bce4c1b2-c47c-4513-9d3e-45292ff415f3_isrForCrosstalkSources_1125894_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/bce4c1b2-c47c-4513-9d3e-45292ff415f3_isrForCrosstalkSources_1125894_47.stderr parsl.dataflow.dflow DEBUG: Task 3594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3594 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3594 has memoization hash bb0b9a6ad0adc4a23e2cf73ac18066c8 parsl.dataflow.memoization INFO: Task 3594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390af9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3594 try 0 launched on executor multi with executor id 3595 parsl.dataflow.dflow INFO: Standard output for task 3594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/14d62464-14cd-43b2-b780-d331cf794127_isrForCrosstalkSources_1125900_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/14d62464-14cd-43b2-b780-d331cf794127_isrForCrosstalkSources_1125900_29.stderr parsl.dataflow.dflow DEBUG: Task 3595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3595 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3595 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3595 has memoization hash 460d8f81cbcc9d545c99233d3d3f1a27 parsl.dataflow.memoization INFO: Task 3595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ad6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3595 try 0 launched on executor multi with executor id 3596 parsl.dataflow.dflow INFO: Standard output for task 3595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d07d3114-bf6b-4668-b6bc-b79f19068b49_isrForCrosstalkSources_1125940_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d07d3114-bf6b-4668-b6bc-b79f19068b49_isrForCrosstalkSources_1125940_4.stderr parsl.dataflow.dflow DEBUG: Task 3596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3596 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3596 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3596 has memoization hash cd6e4a0fc567ad98dff471fe345ed0b9 parsl.dataflow.memoization INFO: Task 3596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ad260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3596 try 0 launched on executor multi with executor id 3597 parsl.dataflow.dflow INFO: Standard output for task 3596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/70c4a5e5-9caf-4c91-a438-73385057da1a_isrForCrosstalkSources_1125928_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/70c4a5e5-9caf-4c91-a438-73385057da1a_isrForCrosstalkSources_1125928_33.stderr parsl.dataflow.dflow DEBUG: Task 3597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3597 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3597 has memoization hash 4194eb149b84a5a09a42bc28836810d9 parsl.dataflow.memoization INFO: Task 3597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ac0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3597 try 0 launched on executor multi with executor id 3598 parsl.dataflow.dflow INFO: Standard output for task 3597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f8743036-924f-4be1-8b7e-2d7daafeff7a_isrForCrosstalkSources_1125945_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f8743036-924f-4be1-8b7e-2d7daafeff7a_isrForCrosstalkSources_1125945_40.stderr parsl.dataflow.dflow DEBUG: Task 3598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3598 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3598 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3598 has memoization hash dea231b50edc83ae4d991b1d01eb1eec parsl.dataflow.memoization INFO: Task 3598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ad300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3598 try 0 launched on executor multi with executor id 3599 parsl.dataflow.dflow INFO: Standard output for task 3598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a5b3f2ff-663e-4a9d-89c0-f68a9cf929ae_isrForCrosstalkSources_1125945_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a5b3f2ff-663e-4a9d-89c0-f68a9cf929ae_isrForCrosstalkSources_1125945_48.stderr parsl.dataflow.dflow DEBUG: Task 3599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3599 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3599 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3599 has memoization hash 6e3abe0daf3ea12efa71b387fb218536 parsl.dataflow.memoization INFO: Task 3599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ac040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3599 try 0 launched on executor multi with executor id 3600 parsl.dataflow.dflow INFO: Standard output for task 3599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/b5cd5194-e348-448a-9ecc-2b2017047a46_isrForCrosstalkSources_1125897_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/b5cd5194-e348-448a-9ecc-2b2017047a46_isrForCrosstalkSources_1125897_5.stderr parsl.dataflow.dflow DEBUG: Task 3600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3600 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3600 has memoization hash bc8b6954f4f034945d407c91454ca30a parsl.dataflow.memoization INFO: Task 3600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ad1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3600 try 0 launched on executor multi with executor id 3601 parsl.dataflow.dflow INFO: Standard output for task 3600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/b5c2dff2-74ae-403a-a3e9-323840277cc7_isrForCrosstalkSources_1125898_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/b5c2dff2-74ae-403a-a3e9-323840277cc7_isrForCrosstalkSources_1125898_48.stderr parsl.dataflow.dflow DEBUG: Task 3601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3601 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3601 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3601 has memoization hash 2e02999bbd1cf9b7a028eaebafbccce2 parsl.dataflow.memoization INFO: Task 3601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ac180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3601 try 0 launched on executor multi with executor id 3602 parsl.dataflow.dflow INFO: Standard output for task 3601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f3e7a046-fb61-451a-99af-df1e661355bb_isrForCrosstalkSources_1125922_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f3e7a046-fb61-451a-99af-df1e661355bb_isrForCrosstalkSources_1125922_3.stderr parsl.dataflow.dflow DEBUG: Task 3602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3602 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3602 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3602 has memoization hash 3aac311a1518b9f32144da5dd0e25667 parsl.dataflow.memoization INFO: Task 3602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ac2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3602 try 0 launched on executor multi with executor id 3603 parsl.dataflow.dflow INFO: Standard output for task 3602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/80f3fc3a-d3fb-4f35-aff8-bb5f59846500_isrForCrosstalkSources_1125945_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/80f3fc3a-d3fb-4f35-aff8-bb5f59846500_isrForCrosstalkSources_1125945_29.stderr parsl.dataflow.dflow DEBUG: Task 3603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3603 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3603 has memoization hash 5d14b440fad839c54825b88190e6c27c parsl.dataflow.memoization INFO: Task 3603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ad120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3603 try 0 launched on executor multi with executor id 3604 parsl.dataflow.dflow INFO: Standard output for task 3603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/deb427ff-27be-4e77-8abc-5ef6d24b0adf_isrForCrosstalkSources_1125933_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/deb427ff-27be-4e77-8abc-5ef6d24b0adf_isrForCrosstalkSources_1125933_10.stderr parsl.dataflow.dflow DEBUG: Task 3604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3604 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3604 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3604 has memoization hash 38db72af569e6bf971303e3831adddd8 parsl.dataflow.memoization INFO: Task 3604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ad080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3604 try 0 launched on executor multi with executor id 3605 parsl.dataflow.dflow INFO: Standard output for task 3604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/0afe984d-8117-476f-a6bd-af6736c4e354_isrForCrosstalkSources_1125919_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/0afe984d-8117-476f-a6bd-af6736c4e354_isrForCrosstalkSources_1125919_45.stderr parsl.dataflow.dflow DEBUG: Task 3605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3605 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3605 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3605 has memoization hash e67cc9949e1cdd2f16f7ffcb8c5913cc parsl.dataflow.memoization INFO: Task 3605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390acf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3605 try 0 launched on executor multi with executor id 3606 parsl.dataflow.dflow INFO: Standard output for task 3605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/c67ddf6f-4825-4081-b04b-21a46fda9a96_isrForCrosstalkSources_1125898_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/c67ddf6f-4825-4081-b04b-21a46fda9a96_isrForCrosstalkSources_1125898_44.stderr parsl.dataflow.dflow DEBUG: Task 3606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3606 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3606 has memoization hash 542d61080db8b30f259294049a77e9ba parsl.dataflow.memoization INFO: Task 3606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390acfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3606 try 0 launched on executor multi with executor id 3607 parsl.dataflow.dflow INFO: Standard output for task 3606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/73c8f704-fdbd-4ec9-974c-abdb1c58286d_isrForCrosstalkSources_1125902_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/73c8f704-fdbd-4ec9-974c-abdb1c58286d_isrForCrosstalkSources_1125902_47.stderr parsl.dataflow.dflow DEBUG: Task 3607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3607 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3607 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3607 has memoization hash 574f6d2150a8a6d42223d4c07d01d933 parsl.dataflow.memoization INFO: Task 3607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390acea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3607 try 0 launched on executor multi with executor id 3608 parsl.dataflow.dflow INFO: Standard output for task 3607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/22456a59-6c79-4e7e-b31d-009831fb0ad9_isrForCrosstalkSources_1125889_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/22456a59-6c79-4e7e-b31d-009831fb0ad9_isrForCrosstalkSources_1125889_50.stderr parsl.dataflow.dflow DEBUG: Task 3608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3608 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3608 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3608 has memoization hash 05da69b4f5d95b189deb3b35b47408d0 parsl.dataflow.memoization INFO: Task 3608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ace00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3608 try 0 launched on executor multi with executor id 3609 parsl.dataflow.dflow INFO: Standard output for task 3608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/2d6aa942-d853-4111-ad9d-55fe1203b641_isrForCrosstalkSources_1125946_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/2d6aa942-d853-4111-ad9d-55fe1203b641_isrForCrosstalkSources_1125946_36.stderr parsl.dataflow.dflow DEBUG: Task 3609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3609 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3609 has memoization hash f43ed4c4bd2c882d17366a32b5fe1c0f parsl.dataflow.memoization INFO: Task 3609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390accc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3609 try 0 launched on executor multi with executor id 3610 parsl.dataflow.dflow INFO: Standard output for task 3609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/4545329a-05bb-4fc9-aa51-a7e9ee2b607c_isrForCrosstalkSources_1125914_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/4545329a-05bb-4fc9-aa51-a7e9ee2b607c_isrForCrosstalkSources_1125914_49.stderr parsl.dataflow.dflow DEBUG: Task 3610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3610 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3610 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3610 has memoization hash f5e32888c2995a67935a8a41afd0220e parsl.dataflow.memoization INFO: Task 3610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390acb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3610 try 0 launched on executor multi with executor id 3611 parsl.dataflow.dflow INFO: Standard output for task 3610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/a2da7c12-c315-413d-8222-db0f6b9954c7_isrForCrosstalkSources_1125929_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/a2da7c12-c315-413d-8222-db0f6b9954c7_isrForCrosstalkSources_1125929_37.stderr parsl.dataflow.dflow DEBUG: Task 3611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3611 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3611 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3611 has memoization hash 9cca2ba5030fe2ddfaa671175f176037 parsl.dataflow.memoization INFO: Task 3611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390acae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3611 try 0 launched on executor multi with executor id 3612 parsl.dataflow.dflow INFO: Standard output for task 3611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/23eeeda8-4bf2-44a2-aa4f-88d40ead79bd_isrForCrosstalkSources_1125935_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/23eeeda8-4bf2-44a2-aa4f-88d40ead79bd_isrForCrosstalkSources_1125935_25.stderr parsl.dataflow.dflow DEBUG: Task 3612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3612 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3612 has memoization hash 8a6c9e7443c36683a375725a0d7f32a3 parsl.dataflow.memoization INFO: Task 3612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390acd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3612 try 0 launched on executor multi with executor id 3613 parsl.dataflow.dflow INFO: Standard output for task 3612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/b2a6d867-f22d-4e12-94c7-e91da4054e1e_isrForCrosstalkSources_1125911_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/b2a6d867-f22d-4e12-94c7-e91da4054e1e_isrForCrosstalkSources_1125911_41.stderr parsl.dataflow.dflow DEBUG: Task 3613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3613 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3613 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3613 has memoization hash 895aa75dd6f509b31fba1e067f4902aa parsl.dataflow.memoization INFO: Task 3613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390acc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3613 try 0 launched on executor multi with executor id 3614 parsl.dataflow.dflow INFO: Standard output for task 3613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/e9c2ecea-afc9-44d9-a1bc-ca53690777d8_isrForCrosstalkSources_1125903_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/e9c2ecea-afc9-44d9-a1bc-ca53690777d8_isrForCrosstalkSources_1125903_45.stderr parsl.dataflow.dflow DEBUG: Task 3614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3614 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3614 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3614 has memoization hash 1eef6b3eb2bb2ec7618241735854c2e1 parsl.dataflow.memoization INFO: Task 3614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ac9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3614 try 0 launched on executor multi with executor id 3615 parsl.dataflow.dflow INFO: Standard output for task 3614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/d6847986-a66c-4be6-b7b7-3eaf42cf9f7b_isrForCrosstalkSources_1125913_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/d6847986-a66c-4be6-b7b7-3eaf42cf9f7b_isrForCrosstalkSources_1125913_14.stderr parsl.dataflow.dflow DEBUG: Task 3615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3615 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3615 has memoization hash f19e6c89149dcae90fc35b4cf84ab777 parsl.dataflow.memoization INFO: Task 3615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ac860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:10 proc_lsst.multi:170 [INFO] [multi] job_id 16896450 proc_lsst.multi INFO: [multi] job_id 16896450 2024-03-07 15:07:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 15:07:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896450 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896450 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 16896450 parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-07 15:07:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-07 15:07:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 15:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3615 try 0 launched on executor multi with executor id 3616 parsl.dataflow.dflow INFO: Standard output for task 3615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/fe50e467-2315-4593-ab4a-bbaa7117b20f_isrForCrosstalkSources_1125918_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/fe50e467-2315-4593-ab4a-bbaa7117b20f_isrForCrosstalkSources_1125918_16.stderr parsl.dataflow.dflow DEBUG: Task 3616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3616 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3616 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3616 has memoization hash 11b98d981f16b53ff968c4dea89e2f10 parsl.dataflow.memoization INFO: Task 3616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390aca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3616 try 0 launched on executor multi with executor id 3617 parsl.dataflow.dflow INFO: Standard output for task 3616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/eebe45a1-37fb-4b4d-87d9-502080a38445_isrForCrosstalkSources_1125930_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/eebe45a1-37fb-4b4d-87d9-502080a38445_isrForCrosstalkSources_1125930_28.stderr parsl.dataflow.dflow DEBUG: Task 3617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3617 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3617 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3617 has memoization hash cd135e512771bda972e29122cb92c13c parsl.dataflow.memoization INFO: Task 3617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390aff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3617 try 0 launched on executor multi with executor id 3618 parsl.dataflow.dflow INFO: Standard output for task 3617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/71e69eaa-c24f-4a18-831f-c02967950e0d_isrForCrosstalkSources_1125890_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/71e69eaa-c24f-4a18-831f-c02967950e0d_isrForCrosstalkSources_1125890_7.stderr parsl.dataflow.dflow DEBUG: Task 3618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3618 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3618 has memoization hash 119bab941c550dd995f8eae57d30cc6d parsl.dataflow.memoization INFO: Task 3618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390afec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3618 try 0 launched on executor multi with executor id 3619 parsl.dataflow.dflow INFO: Standard output for task 3618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/09dfc302-c179-4c0f-9c30-b6ffd2335e70_isrForCrosstalkSources_1125937_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/09dfc302-c179-4c0f-9c30-b6ffd2335e70_isrForCrosstalkSources_1125937_2.stderr parsl.dataflow.dflow DEBUG: Task 3619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3619 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3619 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3619 has memoization hash c4d23728f22860bd1db04b4599e1e331 parsl.dataflow.memoization INFO: Task 3619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ac720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3619 try 0 launched on executor multi with executor id 3620 parsl.dataflow.dflow INFO: Standard output for task 3619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/a6857d9c-e50f-441e-92e4-4ead25991dd6_isrForCrosstalkSources_1125942_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/a6857d9c-e50f-441e-92e4-4ead25991dd6_isrForCrosstalkSources_1125942_46.stderr parsl.dataflow.dflow DEBUG: Task 3620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3620 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3620 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3620 has memoization hash ef21e2dbd9b9fd79d772a4da5c76154e parsl.dataflow.memoization INFO: Task 3620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390afd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3620 try 0 launched on executor multi with executor id 3621 parsl.dataflow.dflow INFO: Standard output for task 3620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0eb050a1-942b-4e11-a022-a60bb4294e6d_isrForCrosstalkSources_1125914_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0eb050a1-942b-4e11-a022-a60bb4294e6d_isrForCrosstalkSources_1125914_28.stderr parsl.dataflow.dflow DEBUG: Task 3621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3621 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3621 has memoization hash aae237690e049ff9d6a10a2e8d67de20 parsl.dataflow.memoization INFO: Task 3621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ad800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3621 try 0 launched on executor multi with executor id 3622 parsl.dataflow.dflow INFO: Standard output for task 3621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/baef05cc-566f-4907-ad0b-afa265827692_isrForCrosstalkSources_1125925_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/baef05cc-566f-4907-ad0b-afa265827692_isrForCrosstalkSources_1125925_44.stderr parsl.dataflow.dflow DEBUG: Task 3622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3622 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3622 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3622 has memoization hash d40af925dbf3dcf8c5f94da2fe7b6a53 parsl.dataflow.memoization INFO: Task 3622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ae480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3622 try 0 launched on executor multi with executor id 3623 parsl.dataflow.dflow INFO: Standard output for task 3622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/1e5f09a2-a4cd-485c-9faa-8e614248fc14_isrForCrosstalkSources_1125898_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/1e5f09a2-a4cd-485c-9faa-8e614248fc14_isrForCrosstalkSources_1125898_22.stderr parsl.dataflow.dflow DEBUG: Task 3623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3623 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3623 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3623 has memoization hash f78195ac5fb9a6a7e74d0f161ea91d1d parsl.dataflow.memoization INFO: Task 3623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ac5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3623 try 0 launched on executor multi with executor id 3624 parsl.dataflow.dflow INFO: Standard output for task 3623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/70a7aed6-c7d5-4498-aef8-bb566152ba26_isrForCrosstalkSources_1125931_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/70a7aed6-c7d5-4498-aef8-bb566152ba26_isrForCrosstalkSources_1125931_49.stderr parsl.dataflow.dflow DEBUG: Task 3624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3624 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3624 has memoization hash 34727018fa9f70218f0c3a1de184e450 parsl.dataflow.memoization INFO: Task 3624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ae3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3624 try 0 launched on executor multi with executor id 3625 parsl.dataflow.dflow INFO: Standard output for task 3624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/b0b80693-8b64-4f70-8d68-5c9965de6359_isrForCrosstalkSources_1125935_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/b0b80693-8b64-4f70-8d68-5c9965de6359_isrForCrosstalkSources_1125935_27.stderr parsl.dataflow.dflow DEBUG: Task 3625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3625 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3625 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3625 has memoization hash 781777ccfa8bed865a80f14b1330003f parsl.dataflow.memoization INFO: Task 3625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ac680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3625 try 0 launched on executor multi with executor id 3626 parsl.dataflow.dflow INFO: Standard output for task 3625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/1e787b0b-d6a5-434a-a911-810bad0e64ad_isrForCrosstalkSources_1125891_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/1e787b0b-d6a5-434a-a911-810bad0e64ad_isrForCrosstalkSources_1125891_47.stderr parsl.dataflow.dflow DEBUG: Task 3626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3626 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3626 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3626 has memoization hash e4241df9a2c4153c2a453a6309a594c2 parsl.dataflow.memoization INFO: Task 3626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ac7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3626 try 0 launched on executor multi with executor id 3627 parsl.dataflow.dflow INFO: Standard output for task 3626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/4e48b53f-df0c-4d8d-8815-a0826550928d_isrForCrosstalkSources_1125915_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/4e48b53f-df0c-4d8d-8815-a0826550928d_isrForCrosstalkSources_1125915_27.stderr parsl.dataflow.dflow DEBUG: Task 3627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3627 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3627 has memoization hash 8ec5631eea4be31e4a7aa01e7425bee1 parsl.dataflow.memoization INFO: Task 3627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ac900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3627 try 0 launched on executor multi with executor id 3628 parsl.dataflow.dflow INFO: Standard output for task 3627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/4dd67bfa-9c09-47ea-b5e1-c1e471a61fa8_isrForCrosstalkSources_1125913_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/4dd67bfa-9c09-47ea-b5e1-c1e471a61fa8_isrForCrosstalkSources_1125913_60.stderr parsl.dataflow.dflow DEBUG: Task 3628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3628 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3628 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3628 has memoization hash d5813754b6a6435f0f167326d5503b52 parsl.dataflow.memoization INFO: Task 3628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ac540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3628 try 0 launched on executor multi with executor id 3629 parsl.dataflow.dflow INFO: Standard output for task 3628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/c8e06794-95b9-4cf2-996b-e3fa2890d74a_isrForCrosstalkSources_1125940_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/c8e06794-95b9-4cf2-996b-e3fa2890d74a_isrForCrosstalkSources_1125940_16.stderr parsl.dataflow.dflow DEBUG: Task 3629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3629 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3629 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3629 has memoization hash 1fc8ebbb532711267a1f274104c66ff2 parsl.dataflow.memoization INFO: Task 3629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ac4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3629 try 0 launched on executor multi with executor id 3630 parsl.dataflow.dflow INFO: Standard output for task 3629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/a6eedaeb-46a6-48a4-a07d-a80a6076f9c6_isrForCrosstalkSources_1125915_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/a6eedaeb-46a6-48a4-a07d-a80a6076f9c6_isrForCrosstalkSources_1125915_38.stderr parsl.dataflow.dflow DEBUG: Task 3630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3630 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3630 has memoization hash be8e41c7702ef99f80c107b2325edc7d parsl.dataflow.memoization INFO: Task 3630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ae520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3630 try 0 launched on executor multi with executor id 3631 parsl.dataflow.dflow INFO: Standard output for task 3630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/ebed7e01-54de-45e7-87c0-79b830c08f9d_isrForCrosstalkSources_1125924_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/ebed7e01-54de-45e7-87c0-79b830c08f9d_isrForCrosstalkSources_1125924_19.stderr parsl.dataflow.dflow DEBUG: Task 3631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3631 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3631 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3631 has memoization hash 430475b10e1d9796b534316af46733c7 parsl.dataflow.memoization INFO: Task 3631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ae340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3631 try 0 launched on executor multi with executor id 3632 parsl.dataflow.dflow INFO: Standard output for task 3631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/03f2d816-872d-4def-9476-bf740695ef9b_isrForCrosstalkSources_1125909_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/03f2d816-872d-4def-9476-bf740695ef9b_isrForCrosstalkSources_1125909_43.stderr parsl.dataflow.dflow DEBUG: Task 3632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3632 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3632 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3632 has memoization hash 3b499e31c30080d8c831b17a84c6df87 parsl.dataflow.memoization INFO: Task 3632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ae200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3632 try 0 launched on executor multi with executor id 3633 parsl.dataflow.dflow INFO: Standard output for task 3632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a0f0f3c3-5308-4747-838d-f7af5a9064b5_isrForCrosstalkSources_1125900_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a0f0f3c3-5308-4747-838d-f7af5a9064b5_isrForCrosstalkSources_1125900_38.stderr parsl.dataflow.dflow DEBUG: Task 3633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3633 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3633 has memoization hash 4ea6defd3dbf013522d89cdcb3087692 parsl.dataflow.memoization INFO: Task 3633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ae2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3633 try 0 launched on executor multi with executor id 3634 parsl.dataflow.dflow INFO: Standard output for task 3633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d5114e4b-7905-462e-9579-80dd187b6d72_isrForCrosstalkSources_1125940_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d5114e4b-7905-462e-9579-80dd187b6d72_isrForCrosstalkSources_1125940_26.stderr parsl.dataflow.dflow DEBUG: Task 3634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3634 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3634 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3634 has memoization hash 1a28d91a5fe283e9d160a3c548afbb83 parsl.dataflow.memoization INFO: Task 3634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ac360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3634 try 0 launched on executor multi with executor id 3635 parsl.dataflow.dflow INFO: Standard output for task 3634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/ce518090-f2d7-42a5-a8f2-cee7c58269c2_isrForCrosstalkSources_1125911_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/ce518090-f2d7-42a5-a8f2-cee7c58269c2_isrForCrosstalkSources_1125911_40.stderr parsl.dataflow.dflow DEBUG: Task 3635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3635 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3635 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3635 has memoization hash ab273535f9e40f3365e4a628735a8b81 parsl.dataflow.memoization INFO: Task 3635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ae0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3635 try 0 launched on executor multi with executor id 3636 parsl.dataflow.dflow INFO: Standard output for task 3635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/79f9f2f0-babb-4e81-a09d-b1f77523cafa_isrForCrosstalkSources_1125913_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/79f9f2f0-babb-4e81-a09d-b1f77523cafa_isrForCrosstalkSources_1125913_23.stderr parsl.dataflow.dflow DEBUG: Task 3636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3636 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3636 has memoization hash d889c6d1c6ff0823cb824ab5acc55634 parsl.dataflow.memoization INFO: Task 3636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ae160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3636 try 0 launched on executor multi with executor id 3637 parsl.dataflow.dflow INFO: Standard output for task 3636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/0381c06e-45f0-49f3-a593-78993afdef87_isrForCrosstalkSources_1125909_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/0381c06e-45f0-49f3-a593-78993afdef87_isrForCrosstalkSources_1125909_48.stderr parsl.dataflow.dflow DEBUG: Task 3637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3637 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3637 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3637 has memoization hash ffeea7c3f64ec5860d2688432db6a8ff parsl.dataflow.memoization INFO: Task 3637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390adee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:10 proc_lsst.multi:170 [INFO] [multi] job_id 16896451 proc_lsst.multi INFO: [multi] job_id 16896451 2024-03-07 15:07:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 15:07:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896451 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896451 parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 16896451 parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-07 15:07:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-07 15:07:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 15:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3637 try 0 launched on executor multi with executor id 3638 parsl.dataflow.dflow INFO: Standard output for task 3637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/898914bd-234d-4c0b-a689-74bc26209ea7_isrForCrosstalkSources_1125888_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/898914bd-234d-4c0b-a689-74bc26209ea7_isrForCrosstalkSources_1125888_47.stderr parsl.dataflow.dflow DEBUG: Task 3638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3638 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3638 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3638 has memoization hash cf232f992c5d8397f9dc14412c64e58a parsl.dataflow.memoization INFO: Task 3638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ae020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3638 try 0 launched on executor multi with executor id 3639 parsl.dataflow.dflow INFO: Standard output for task 3638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/9b951d20-9266-479d-9d01-0573edc46215_isrForCrosstalkSources_1125916_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/9b951d20-9266-479d-9d01-0573edc46215_isrForCrosstalkSources_1125916_15.stderr parsl.dataflow.dflow DEBUG: Task 3639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3639 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3639 has memoization hash 1bbe80de6f2f9961f9d4eb78c4a0e854 parsl.dataflow.memoization INFO: Task 3639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390afce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3639 try 0 launched on executor multi with executor id 3640 parsl.dataflow.dflow INFO: Standard output for task 3639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/86955407-6ac5-4181-aecf-93c37572f2f6_isrForCrosstalkSources_1125939_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/86955407-6ac5-4181-aecf-93c37572f2f6_isrForCrosstalkSources_1125939_51.stderr parsl.dataflow.dflow DEBUG: Task 3640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3640 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3640 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3640 has memoization hash 276a67f2223c8f327a85405339f6dc24 parsl.dataflow.memoization INFO: Task 3640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390add00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3640 try 0 launched on executor multi with executor id 3641 parsl.dataflow.dflow INFO: Standard output for task 3640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/29b7c81b-9855-4faf-b976-fd902394c1a6_isrForCrosstalkSources_1125916_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/29b7c81b-9855-4faf-b976-fd902394c1a6_isrForCrosstalkSources_1125916_21.stderr parsl.dataflow.dflow DEBUG: Task 3641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3641 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3641 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3641 has memoization hash 6aa7d1f0977d22cac47fa9b1472005fe parsl.dataflow.memoization INFO: Task 3641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3641 try 0 launched on executor multi with executor id 3642 parsl.dataflow.dflow INFO: Standard output for task 3641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/877d790f-0ef5-43a5-9c42-e6ddbf5681e2_isrForCrosstalkSources_1125919_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/877d790f-0ef5-43a5-9c42-e6ddbf5681e2_isrForCrosstalkSources_1125919_11.stderr parsl.dataflow.dflow DEBUG: Task 3642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3642 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3642 has memoization hash 01c6bede693ef8eb8054673db08eb85d parsl.dataflow.memoization INFO: Task 3642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390afe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3642 try 0 launched on executor multi with executor id 3643 parsl.dataflow.dflow INFO: Standard output for task 3642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/ce521ac3-1c57-4f4e-8b9d-01f1f059648b_isrForCrosstalkSources_1125907_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/ce521ac3-1c57-4f4e-8b9d-01f1f059648b_isrForCrosstalkSources_1125907_27.stderr parsl.dataflow.dflow DEBUG: Task 3643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3643 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3643 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3643 has memoization hash 22ab1ca65e8f7ea90d0bc9a9717b02df parsl.dataflow.memoization INFO: Task 3643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3643 try 0 launched on executor multi with executor id 3644 parsl.dataflow.dflow INFO: Standard output for task 3643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d10386d0-d768-47cc-95e5-a5fe0df6609e_isrForCrosstalkSources_1125915_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d10386d0-d768-47cc-95e5-a5fe0df6609e_isrForCrosstalkSources_1125915_5.stderr parsl.dataflow.dflow DEBUG: Task 3644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3644 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3644 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3644 has memoization hash 5a42c592de3c47a4ce2d60c29ee02fd5 parsl.dataflow.memoization INFO: Task 3644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3644 try 0 launched on executor multi with executor id 3645 parsl.dataflow.dflow INFO: Standard output for task 3644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/01eab606-2192-4c72-894d-b8aa67f41ba7_isrForCrosstalkSources_1125917_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/01eab606-2192-4c72-894d-b8aa67f41ba7_isrForCrosstalkSources_1125917_37.stderr parsl.dataflow.dflow DEBUG: Task 3645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3645 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3645 has memoization hash 5a0b686af38ab9346bddb338abea42a0 parsl.dataflow.memoization INFO: Task 3645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390adb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3645 try 0 launched on executor multi with executor id 3646 parsl.dataflow.dflow INFO: Standard output for task 3645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/2f3bb024-9f8c-4fcf-a4c9-58d40e06e0bc_isrForCrosstalkSources_1125917_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/2f3bb024-9f8c-4fcf-a4c9-58d40e06e0bc_isrForCrosstalkSources_1125917_11.stderr parsl.dataflow.dflow DEBUG: Task 3646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3646 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3646 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3646 has memoization hash 3c9469108f1d5701ced0b620ab2c54a0 parsl.dataflow.memoization INFO: Task 3646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390adc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3646 try 0 launched on executor multi with executor id 3647 parsl.dataflow.dflow INFO: Standard output for task 3646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/91c76eeb-bc7f-429f-9b42-d982f9120a26_isrForCrosstalkSources_1125890_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/91c76eeb-bc7f-429f-9b42-d982f9120a26_isrForCrosstalkSources_1125890_10.stderr parsl.dataflow.dflow DEBUG: Task 3647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3647 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3647 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3647 has memoization hash 808d920009e427cd65725fe5a5060803 parsl.dataflow.memoization INFO: Task 3647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390adbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3647 try 0 launched on executor multi with executor id 3648 parsl.dataflow.dflow INFO: Standard output for task 3647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/51bc3091-db89-4033-b9da-042790a1fd98_isrForCrosstalkSources_1125899_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/51bc3091-db89-4033-b9da-042790a1fd98_isrForCrosstalkSources_1125899_18.stderr parsl.dataflow.dflow DEBUG: Task 3648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3648 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3648 has memoization hash dd597d238ddc57874f6969942f1ed50e parsl.dataflow.memoization INFO: Task 3648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ada80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3648 try 0 launched on executor multi with executor id 3649 parsl.dataflow.dflow INFO: Standard output for task 3648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/e4ebe6fa-c0c9-4540-94b3-19474f8bb510_isrForCrosstalkSources_1125898_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/e4ebe6fa-c0c9-4540-94b3-19474f8bb510_isrForCrosstalkSources_1125898_62.stderr parsl.dataflow.dflow DEBUG: Task 3649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3649 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3649 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3649 has memoization hash 45f291f234485d7f6808cd25baced51f parsl.dataflow.memoization INFO: Task 3649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ad9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3649 try 0 launched on executor multi with executor id 3650 parsl.dataflow.dflow INFO: Standard output for task 3649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/42467f8a-3a79-4390-99a9-cd0bce27c2b9_isrForCrosstalkSources_1125933_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/42467f8a-3a79-4390-99a9-cd0bce27c2b9_isrForCrosstalkSources_1125933_48.stderr parsl.dataflow.dflow DEBUG: Task 3650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3650 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3650 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3650 has memoization hash 32999bec2068d450c48ddc590e3898d3 parsl.dataflow.memoization INFO: Task 3650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ad940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3650 try 0 launched on executor multi with executor id 3651 parsl.dataflow.dflow INFO: Standard output for task 3650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/dfd97ceb-f4d0-4716-9e32-e73a9590666c_isrForCrosstalkSources_1125919_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/dfd97ceb-f4d0-4716-9e32-e73a9590666c_isrForCrosstalkSources_1125919_3.stderr parsl.dataflow.dflow DEBUG: Task 3651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3651 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3651 has memoization hash 5aad89c8a3e84d68f04a8c5f9817047b parsl.dataflow.memoization INFO: Task 3651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ad760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3651 try 0 launched on executor multi with executor id 3652 parsl.dataflow.dflow INFO: Standard output for task 3651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/513c862f-759f-4a3d-a751-3b0139c92d58_isrForCrosstalkSources_1125935_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/513c862f-759f-4a3d-a751-3b0139c92d58_isrForCrosstalkSources_1125935_38.stderr parsl.dataflow.dflow DEBUG: Task 3652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3652 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3652 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3652 has memoization hash 47b9b8dfbfa36dd72b946be434c98b5b parsl.dataflow.memoization INFO: Task 3652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390af920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3652 try 0 launched on executor multi with executor id 3653 parsl.dataflow.dflow INFO: Standard output for task 3652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/7fbffb57-f32c-429e-9960-d35102216a46_isrForCrosstalkSources_1125920_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/7fbffb57-f32c-429e-9960-d35102216a46_isrForCrosstalkSources_1125920_29.stderr parsl.dataflow.dflow DEBUG: Task 3653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3653 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3653 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3653 has memoization hash eaaf3c2899f1d4d7b028c325b22a5e02 parsl.dataflow.memoization INFO: Task 3653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ad8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3653 try 0 launched on executor multi with executor id 3654 parsl.dataflow.dflow INFO: Standard output for task 3653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/ad3383e1-f648-44bc-b601-0e0080853950_isrForCrosstalkSources_1125930_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/ad3383e1-f648-44bc-b601-0e0080853950_isrForCrosstalkSources_1125930_15.stderr parsl.dataflow.dflow DEBUG: Task 3654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3654 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3654 has memoization hash ca6339259e86167865c5479c42b593c1 parsl.dataflow.memoization INFO: Task 3654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390af7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3654 try 0 launched on executor multi with executor id 3655 parsl.dataflow.dflow INFO: Standard output for task 3654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f481f751-3ddb-4f6c-96d4-7d108a9803c1_isrForCrosstalkSources_1125939_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f481f751-3ddb-4f6c-96d4-7d108a9803c1_isrForCrosstalkSources_1125939_26.stderr parsl.dataflow.dflow DEBUG: Task 3655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3655 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3655 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3655 has memoization hash df64a31d3012a38df2336b393907d180 parsl.dataflow.memoization INFO: Task 3655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390af880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3655 try 0 launched on executor multi with executor id 3656 parsl.dataflow.dflow INFO: Standard output for task 3655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/18a16579-2980-4923-a15b-8fe24685d643_isrForCrosstalkSources_1125932_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/18a16579-2980-4923-a15b-8fe24685d643_isrForCrosstalkSources_1125932_39.stderr parsl.dataflow.dflow DEBUG: Task 3656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3656 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3656 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3656 has memoization hash 14e58f85acc27ad22132b8bcb28ae50b parsl.dataflow.memoization INFO: Task 3656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390af6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3656 try 0 launched on executor multi with executor id 3657 parsl.dataflow.dflow INFO: Standard output for task 3656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/c2e919b2-fa2c-4cd8-9164-ab479837b875_isrForCrosstalkSources_1125894_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/c2e919b2-fa2c-4cd8-9164-ab479837b875_isrForCrosstalkSources_1125894_56.stderr parsl.dataflow.dflow DEBUG: Task 3657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3657 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3657 has memoization hash 7eddbe1d8ec3bc7c5b873a9b7951c8b4 parsl.dataflow.memoization INFO: Task 3657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390af560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3657 try 0 launched on executor multi with executor id 3658 parsl.dataflow.dflow INFO: Standard output for task 3657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/32c64b8a-7b4c-4df0-8e51-af73c3cc6fda_isrForCrosstalkSources_1125893_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/32c64b8a-7b4c-4df0-8e51-af73c3cc6fda_isrForCrosstalkSources_1125893_45.stderr parsl.dataflow.dflow DEBUG: Task 3658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3658 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3658 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3658 has memoization hash 8704145e09936a9280bfc8a919ab825e parsl.dataflow.memoization INFO: Task 3658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390af740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3658 try 0 launched on executor multi with executor id 3659 parsl.dataflow.dflow INFO: Standard output for task 3658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/50f96941-64e9-411d-bc5c-f2a7683b566a_isrForCrosstalkSources_1125942_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/50f96941-64e9-411d-bc5c-f2a7683b566a_isrForCrosstalkSources_1125942_33.stderr parsl.dataflow.dflow DEBUG: Task 3659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3659 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3659 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3659 has memoization hash 17c681e5254d821fd81d60d95432cad6 parsl.dataflow.memoization INFO: Task 3659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390af420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3659 try 0 launched on executor multi with executor id 3660 parsl.dataflow.dflow INFO: Standard output for task 3659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/4bef2bef-e47f-4f7a-b293-51009c896389_isrForCrosstalkSources_1125917_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/4bef2bef-e47f-4f7a-b293-51009c896389_isrForCrosstalkSources_1125917_23.stderr parsl.dataflow.dflow DEBUG: Task 3660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3660 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3660 has memoization hash f1da32b89648890cabe47b3465f73b80 parsl.dataflow.memoization INFO: Task 3660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390af380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3660 try 0 launched on executor multi with executor id 3661 parsl.dataflow.dflow INFO: Standard output for task 3660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/dcfd9a02-7dc7-4f52-8f11-4b728faac4a6_isrForCrosstalkSources_1125941_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/dcfd9a02-7dc7-4f52-8f11-4b728faac4a6_isrForCrosstalkSources_1125941_55.stderr parsl.dataflow.dflow DEBUG: Task 3661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3661 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3661 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3661 has memoization hash b1f5a5f104828dad49eaa60b0ed15b4b parsl.dataflow.memoization INFO: Task 3661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390af4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3661 try 0 launched on executor multi with executor id 3662 parsl.dataflow.dflow INFO: Standard output for task 3661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/ac99be26-18ac-4032-82d6-46f95a22fdb8_isrForCrosstalkSources_1125936_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/ac99be26-18ac-4032-82d6-46f95a22fdb8_isrForCrosstalkSources_1125936_57.stderr parsl.dataflow.dflow DEBUG: Task 3662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3662 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3662 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3662 has memoization hash af10a41270d6fff8ba5fb819053a2783 parsl.dataflow.memoization INFO: Task 3662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390af240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3662 try 0 launched on executor multi with executor id 3663 parsl.dataflow.dflow INFO: Standard output for task 3662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/6927c681-ea3b-4eef-ab39-a1061181c7e1_isrForCrosstalkSources_1125892_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/6927c681-ea3b-4eef-ab39-a1061181c7e1_isrForCrosstalkSources_1125892_5.stderr parsl.dataflow.dflow DEBUG: Task 3663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3663 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3663 has memoization hash 60337c47c166f224e27fb8f01c247233 parsl.dataflow.memoization INFO: Task 3663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390af600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3663 try 0 launched on executor multi with executor id 3664 parsl.dataflow.dflow INFO: Standard output for task 3663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d60a0674-b9c1-4647-a7e1-d2db9dae0d0b_isrForCrosstalkSources_1125919_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d60a0674-b9c1-4647-a7e1-d2db9dae0d0b_isrForCrosstalkSources_1125919_26.stderr parsl.dataflow.dflow DEBUG: Task 3664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3664 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3664 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3664 has memoization hash f24ff682f8e064ca64afbc9900393325 parsl.dataflow.memoization INFO: Task 3664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390af100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3664 try 0 launched on executor multi with executor id 3665 parsl.dataflow.dflow INFO: Standard output for task 3664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/f59a94ee-1e10-41eb-b082-57db1eed0fe6_isrForCrosstalkSources_1125918_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/f59a94ee-1e10-41eb-b082-57db1eed0fe6_isrForCrosstalkSources_1125918_48.stderr parsl.dataflow.dflow DEBUG: Task 3665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3665 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3665 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3665 has memoization hash c418dd1ddfd6e85f01ad3d7fa897265e parsl.dataflow.memoization INFO: Task 3665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390af2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:10 proc_lsst.multi:170 [INFO] [multi] job_id 16896452 proc_lsst.multi INFO: [multi] job_id 16896452 2024-03-07 15:07:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 15:07:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896452 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896452 parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 16896452 parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-07 15:07:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-07 15:07:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 15:07:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3665 try 0 launched on executor multi with executor id 3666 parsl.dataflow.dflow INFO: Standard output for task 3665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/e95e886d-3a16-429a-b705-9b93ec2b0a71_isrForCrosstalkSources_1125902_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/e95e886d-3a16-429a-b705-9b93ec2b0a71_isrForCrosstalkSources_1125902_1.stderr parsl.dataflow.dflow DEBUG: Task 3666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3666 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3666 has memoization hash f4684ee1510466517ea860fd7f3e5ecf parsl.dataflow.memoization INFO: Task 3666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390aefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3666 try 0 launched on executor multi with executor id 3667 parsl.dataflow.dflow INFO: Standard output for task 3666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/b3753702-b34e-4927-9cc4-62a154d2961a_isrForCrosstalkSources_1125890_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/b3753702-b34e-4927-9cc4-62a154d2961a_isrForCrosstalkSources_1125890_57.stderr parsl.dataflow.dflow DEBUG: Task 3667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3667 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3667 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3667 has memoization hash d607c2428685a3ed61c379ab81a56ab7 parsl.dataflow.memoization INFO: Task 3667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390adf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3667 try 0 launched on executor multi with executor id 3668 parsl.dataflow.dflow INFO: Standard output for task 3667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/3ea3e9b3-1d55-49c5-80c2-650659a7914b_isrForCrosstalkSources_1125898_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/3ea3e9b3-1d55-49c5-80c2-650659a7914b_isrForCrosstalkSources_1125898_24.stderr parsl.dataflow.dflow DEBUG: Task 3668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3668 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3668 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3668 has memoization hash d440823dfdfef34d9b18db1d4edb6ef0 parsl.dataflow.memoization INFO: Task 3668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390af1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3668 try 0 launched on executor multi with executor id 3669 parsl.dataflow.dflow INFO: Standard output for task 3668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/3b05418f-cb9c-4e88-a2e5-3a016edcab36_isrForCrosstalkSources_1125919_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/3b05418f-cb9c-4e88-a2e5-3a016edcab36_isrForCrosstalkSources_1125919_31.stderr parsl.dataflow.dflow DEBUG: Task 3669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3669 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3669 has memoization hash 435a263d7ec723590a27b7c55d20547e parsl.dataflow.memoization INFO: Task 3669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390adda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3669 try 0 launched on executor multi with executor id 3670 parsl.dataflow.dflow INFO: Standard output for task 3669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/74b7d879-dd83-4da3-b609-4a055cc2b250_isrForCrosstalkSources_1125915_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/74b7d879-dd83-4da3-b609-4a055cc2b250_isrForCrosstalkSources_1125915_59.stderr parsl.dataflow.dflow DEBUG: Task 3670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3670 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3670 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3670 has memoization hash 9e7a291677616b715060b8b7d9e9cc81 parsl.dataflow.memoization INFO: Task 3670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390aed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3670 try 0 launched on executor multi with executor id 3671 parsl.dataflow.dflow INFO: Standard output for task 3670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/e2bc822f-29c2-4cad-85bd-41d63fd0f9f0_isrForCrosstalkSources_1125900_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/e2bc822f-29c2-4cad-85bd-41d63fd0f9f0_isrForCrosstalkSources_1125900_2.stderr parsl.dataflow.dflow DEBUG: Task 3671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3671 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3671 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3671 has memoization hash 84d2c336370b87a9dbcfa24641447cfb parsl.dataflow.memoization INFO: Task 3671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ac220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3671 try 0 launched on executor multi with executor id 3672 parsl.dataflow.dflow INFO: Standard output for task 3671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/c15bf00c-0958-44af-a767-76c4f4b059e8_isrForCrosstalkSources_1125929_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/c15bf00c-0958-44af-a767-76c4f4b059e8_isrForCrosstalkSources_1125929_30.stderr parsl.dataflow.dflow DEBUG: Task 3672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3672 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3672 has memoization hash 4418a3bd101d29b52bb7208456ff9d0b parsl.dataflow.memoization INFO: Task 3672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390aec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3672 try 0 launched on executor multi with executor id 3673 parsl.dataflow.dflow INFO: Standard output for task 3672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/df9330ee-6d6e-4059-a69a-dfe5b0aa9c94_isrForCrosstalkSources_1125912_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/df9330ee-6d6e-4059-a69a-dfe5b0aa9c94_isrForCrosstalkSources_1125912_43.stderr parsl.dataflow.dflow DEBUG: Task 3673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3673 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3673 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3673 has memoization hash 462f92662b19457f3bc40f0e1178ffc1 parsl.dataflow.memoization INFO: Task 3673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390af060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3673 try 0 launched on executor multi with executor id 3674 parsl.dataflow.dflow INFO: Standard output for task 3673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/1d695cf5-5b72-48e6-8c9e-7bfd7246b85f_isrForCrosstalkSources_1125944_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/1d695cf5-5b72-48e6-8c9e-7bfd7246b85f_isrForCrosstalkSources_1125944_6.stderr parsl.dataflow.dflow DEBUG: Task 3674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3674 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3674 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3674 has memoization hash bd8aabfe6dc64a4e103bae6dd87144d9 parsl.dataflow.memoization INFO: Task 3674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ac400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3674 try 0 launched on executor multi with executor id 3675 parsl.dataflow.dflow INFO: Standard output for task 3674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f2542e42-21bd-4b97-848a-6e7f2704e2ad_isrForCrosstalkSources_1125897_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f2542e42-21bd-4b97-848a-6e7f2704e2ad_isrForCrosstalkSources_1125897_57.stderr parsl.dataflow.dflow DEBUG: Task 3675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3675 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3675 has memoization hash 7b1e1057310dd2ff36b2a7cd6d14751f parsl.dataflow.memoization INFO: Task 3675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390aede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3675 try 0 launched on executor multi with executor id 3676 parsl.dataflow.dflow INFO: Standard output for task 3675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/249c871d-0a00-417a-a66c-584724a753c7_isrForCrosstalkSources_1125904_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/249c871d-0a00-417a-a66c-584724a753c7_isrForCrosstalkSources_1125904_24.stderr parsl.dataflow.dflow DEBUG: Task 3676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3676 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3676 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3676 has memoization hash 2f8ae665caf9ce2b504dc88df466bb2c parsl.dataflow.memoization INFO: Task 3676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390aeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3676 try 0 launched on executor multi with executor id 3677 parsl.dataflow.dflow INFO: Standard output for task 3676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/872b128a-abca-47c2-80f1-4a4221d4c176_isrForCrosstalkSources_1125898_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/872b128a-abca-47c2-80f1-4a4221d4c176_isrForCrosstalkSources_1125898_36.stderr parsl.dataflow.dflow DEBUG: Task 3677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3677 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3677 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3677 has memoization hash 0519f5a71c09fd499e29f63f84a10447 parsl.dataflow.memoization INFO: Task 3677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390aeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3677 try 0 launched on executor multi with executor id 3678 parsl.dataflow.dflow INFO: Standard output for task 3677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/87d1c460-ac8d-4799-add6-dc43143202c0_isrForCrosstalkSources_1125915_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/87d1c460-ac8d-4799-add6-dc43143202c0_isrForCrosstalkSources_1125915_43.stderr parsl.dataflow.dflow DEBUG: Task 3678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3678 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3678 has memoization hash e980558ae94bb34468af61300e4d10b0 parsl.dataflow.memoization INFO: Task 3678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390aeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3678 try 0 launched on executor multi with executor id 3679 parsl.dataflow.dflow INFO: Standard output for task 3678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/aadcf80d-4892-4720-ae94-19a508eb6680_isrForCrosstalkSources_1125941_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/aadcf80d-4892-4720-ae94-19a508eb6680_isrForCrosstalkSources_1125941_39.stderr parsl.dataflow.dflow DEBUG: Task 3679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3679 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3679 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3679 has memoization hash 30d6d54a692ea5401b7cf60d0be78fcc parsl.dataflow.memoization INFO: Task 3679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ae980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3679 try 0 launched on executor multi with executor id 3680 parsl.dataflow.dflow INFO: Standard output for task 3679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/bcfa0bef-c4f3-4703-8b4e-4ab4e9d42766_isrForCrosstalkSources_1125918_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/bcfa0bef-c4f3-4703-8b4e-4ab4e9d42766_isrForCrosstalkSources_1125918_15.stderr parsl.dataflow.dflow DEBUG: Task 3680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3680 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3680 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3680 has memoization hash caafc25ba7f396d7728dbcec055bffbe parsl.dataflow.memoization INFO: Task 3680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3680 try 0 launched on executor multi with executor id 3681 parsl.dataflow.dflow INFO: Standard output for task 3680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/f52cf7cc-984c-4732-b902-e8a1496685dd_isrForCrosstalkSources_1125937_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/f52cf7cc-984c-4732-b902-e8a1496685dd_isrForCrosstalkSources_1125937_4.stderr parsl.dataflow.dflow DEBUG: Task 3681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3681 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3681 has memoization hash 87e441ab318c1281d629c45c0ed4d9b5 parsl.dataflow.memoization INFO: Task 3681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3681 try 0 launched on executor multi with executor id 3682 parsl.dataflow.dflow INFO: Standard output for task 3681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/7b27a9b4-f79c-40bb-a5c8-6cbb59a8d9b8_isrForCrosstalkSources_1125890_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/7b27a9b4-f79c-40bb-a5c8-6cbb59a8d9b8_isrForCrosstalkSources_1125890_49.stderr parsl.dataflow.dflow DEBUG: Task 3682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3682 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3682 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3682 has memoization hash 998c0442da34d924abb6bc3e5336bf60 parsl.dataflow.memoization INFO: Task 3682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3682 try 0 launched on executor multi with executor id 3683 parsl.dataflow.dflow INFO: Standard output for task 3682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/c1272209-7fd4-48b2-a275-4c55a1e039ef_isrForCrosstalkSources_1125930_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/c1272209-7fd4-48b2-a275-4c55a1e039ef_isrForCrosstalkSources_1125930_2.stderr parsl.dataflow.dflow DEBUG: Task 3683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3683 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3683 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3683 has memoization hash dbe7a7b9222c45dabb642ee9ad0c1e92 parsl.dataflow.memoization INFO: Task 3683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3683 try 0 launched on executor multi with executor id 3684 parsl.dataflow.dflow INFO: Standard output for task 3683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/c4b1a594-1b8c-465b-9eef-62a7b033aca5_isrForCrosstalkSources_1125915_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/c4b1a594-1b8c-465b-9eef-62a7b033aca5_isrForCrosstalkSources_1125915_7.stderr parsl.dataflow.dflow DEBUG: Task 3684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3684 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3684 has memoization hash c52dccfdde903d54727d17f7655308f6 parsl.dataflow.memoization INFO: Task 3684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3684 try 0 launched on executor multi with executor id 3685 parsl.dataflow.dflow INFO: Standard output for task 3684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/be710b1c-4524-4518-b36c-5bc09c7376c1_isrForCrosstalkSources_1125922_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/be710b1c-4524-4518-b36c-5bc09c7376c1_isrForCrosstalkSources_1125922_41.stderr parsl.dataflow.dflow DEBUG: Task 3685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3685 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3685 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3685 has memoization hash a0d070e1a25147d1cbb220388f996015 parsl.dataflow.memoization INFO: Task 3685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3685 try 0 launched on executor multi with executor id 3686 parsl.dataflow.dflow INFO: Standard output for task 3685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/38853f60-f253-4c61-9361-1e418814bc99_isrForCrosstalkSources_1125920_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/38853f60-f253-4c61-9361-1e418814bc99_isrForCrosstalkSources_1125920_27.stderr parsl.dataflow.dflow DEBUG: Task 3686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3686 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3686 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3686 has memoization hash dce77e4b642e768826c0445ed6837b37 parsl.dataflow.memoization INFO: Task 3686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0cce9ed254d8b52811eae24411050bb with result from task 2 2024-03-07 15:07:11 proc_lsst.multi:170 [INFO] [multi] job_id 16896453 proc_lsst.multi INFO: [multi] job_id 16896453 2024-03-07 15:07:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 15:07:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896453 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896453 parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 16896453 parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-07 15:07:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-07 15:07:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 15:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3686 try 0 launched on executor multi with executor id 3687 parsl.dataflow.dflow INFO: Standard output for task 3686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/76295ad8-393c-4706-b25c-b40439780766_isrForCrosstalkSources_1125909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/76295ad8-393c-4706-b25c-b40439780766_isrForCrosstalkSources_1125909_2.stderr parsl.dataflow.dflow DEBUG: Task 3687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3687 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3687 has memoization hash 5d86a2b96d3d97e977d004be1aa45b80 parsl.dataflow.memoization INFO: Task 3687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390aee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3687 try 0 launched on executor multi with executor id 3688 parsl.dataflow.dflow INFO: Standard output for task 3687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/bcf5f891-c6e1-4cba-a69b-b2a6c91639b7_isrForCrosstalkSources_1125943_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/bcf5f891-c6e1-4cba-a69b-b2a6c91639b7_isrForCrosstalkSources_1125943_19.stderr parsl.dataflow.dflow DEBUG: Task 3688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3688 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3688 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3688 has memoization hash 96c5656af0ce9047dc4feaa3ee95f6b5 parsl.dataflow.memoization INFO: Task 3688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390aef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3688 try 0 launched on executor multi with executor id 3689 parsl.dataflow.dflow INFO: Standard output for task 3688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/f64ae197-7b11-41be-93d4-4556deb2118b_isrForCrosstalkSources_1125947_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/f64ae197-7b11-41be-93d4-4556deb2118b_isrForCrosstalkSources_1125947_28.stderr parsl.dataflow.dflow DEBUG: Task 3689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3689 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3689 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3689 has memoization hash fa913ff793984bd306379da90d7cec4a parsl.dataflow.memoization INFO: Task 3689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a390ade40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 3689 try 0 launched on executor multi with executor id 3690 parsl.dataflow.dflow INFO: Standard output for task 3689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ec1dbc84-cb33-4666-a89d-87553eabcc4e_isrForCrosstalkSources_1125902_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ec1dbc84-cb33-4666-a89d-87553eabcc4e_isrForCrosstalkSources_1125902_50.stderr parsl.dataflow.dflow DEBUG: Task 3690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3690 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3690 has memoization hash f8d0118147bb41063b91e40c878f753c parsl.dataflow.memoization INFO: Task 3690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/dc2fddc1-ce59-410e-9811-87fce004b75b_isrForCrosstalkSources_1125937_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/dc2fddc1-ce59-410e-9811-87fce004b75b_isrForCrosstalkSources_1125937_57.stderr parsl.dataflow.dflow INFO: Parsl task 3690 try 0 launched on executor multi with executor id 3691 parsl.dataflow.dflow INFO: Standard output for task 3690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/9e03135c-1711-4750-a36b-2e375d1e6514_isrForCrosstalkSources_1125940_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/9e03135c-1711-4750-a36b-2e375d1e6514_isrForCrosstalkSources_1125940_39.stderr parsl.dataflow.dflow DEBUG: Task 3691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3691 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3691 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3691 has memoization hash c541f19c4dab15c31d16d8eccb1a92ff parsl.dataflow.memoization INFO: Task 3691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3691 try 0 launched on executor multi with executor id 3692 parsl.dataflow.dflow INFO: Standard output for task 3691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d40cc16d-373e-4412-be5b-7607739f3ef9_isrForCrosstalkSources_1125915_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d40cc16d-373e-4412-be5b-7607739f3ef9_isrForCrosstalkSources_1125915_50.stderr parsl.dataflow.dflow DEBUG: Task 3692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3692 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3692 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3692 has memoization hash 2043040cd4dad92999215b43f631666e parsl.dataflow.memoization INFO: Task 3692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3692 try 0 launched on executor multi with executor id 3693 parsl.dataflow.dflow INFO: Standard output for task 3692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/1c479cd6-7049-4db3-9ceb-d3706a2bb223_isrForCrosstalkSources_1125935_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/1c479cd6-7049-4db3-9ceb-d3706a2bb223_isrForCrosstalkSources_1125935_12.stderr parsl.dataflow.dflow DEBUG: Task 3693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3693 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3693 has memoization hash 2a7c86457df8289b5baf7880fb1efdde parsl.dataflow.memoization INFO: Task 3693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3693 try 0 launched on executor multi with executor id 3694 parsl.dataflow.dflow INFO: Standard output for task 3693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6d5cd192-18b4-4f32-b926-af889e776615_isrForCrosstalkSources_1125927_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6d5cd192-18b4-4f32-b926-af889e776615_isrForCrosstalkSources_1125927_58.stderr parsl.dataflow.dflow DEBUG: Task 3694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3694 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3694 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3694 has memoization hash aa83539beedb56d0a003c8d7cc35befb parsl.dataflow.memoization INFO: Task 3694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3694 try 0 launched on executor multi with executor id 3695 parsl.dataflow.dflow INFO: Standard output for task 3694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/672288d2-cab9-4349-81ec-633febf46563_isrForCrosstalkSources_1125946_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/672288d2-cab9-4349-81ec-633febf46563_isrForCrosstalkSources_1125946_59.stderr parsl.dataflow.dflow DEBUG: Task 3695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3695 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3695 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3695 has memoization hash 03a8d6e260607c98b90e3b6ec20a05ed parsl.dataflow.memoization INFO: Task 3695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3695 try 0 launched on executor multi with executor id 3696 parsl.dataflow.dflow INFO: Standard output for task 3695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/81b039f1-6f17-404f-b999-ed031dd7fd2d_isrForCrosstalkSources_1125889_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/81b039f1-6f17-404f-b999-ed031dd7fd2d_isrForCrosstalkSources_1125889_14.stderr parsl.dataflow.dflow DEBUG: Task 3696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3696 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3696 has memoization hash b2d12062b6ff56009437630a95356f5f parsl.dataflow.memoization INFO: Task 3696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3696 try 0 launched on executor multi with executor id 3697 parsl.dataflow.dflow INFO: Standard output for task 3696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/78953df3-3b44-4b01-a063-f4b96576c7a2_isrForCrosstalkSources_1125892_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/78953df3-3b44-4b01-a063-f4b96576c7a2_isrForCrosstalkSources_1125892_57.stderr parsl.dataflow.dflow DEBUG: Task 3697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3697 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3697 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3697 has memoization hash 0488179b695d1311150b31fd74c79efa parsl.dataflow.memoization INFO: Task 3697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3697 try 0 launched on executor multi with executor id 3698 parsl.dataflow.dflow INFO: Standard output for task 3697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/8b2654fb-3837-4e54-a9a0-8479db6d0d05_isrForCrosstalkSources_1125929_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/8b2654fb-3837-4e54-a9a0-8479db6d0d05_isrForCrosstalkSources_1125929_35.stderr parsl.dataflow.dflow DEBUG: Task 3698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3698 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3698 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3698 has memoization hash 479849bf1e67eb88999d7b35e942b9fa parsl.dataflow.memoization INFO: Task 3698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3698 try 0 launched on executor multi with executor id 3699 parsl.dataflow.dflow INFO: Standard output for task 3698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/2b2ccc1f-fb50-4c4d-a711-42a6be8f1d57_isrForCrosstalkSources_1125901_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/2b2ccc1f-fb50-4c4d-a711-42a6be8f1d57_isrForCrosstalkSources_1125901_4.stderr parsl.dataflow.dflow DEBUG: Task 3699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3699 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3699 has memoization hash 30b2019f00c0171d2d1471f0944fda64 parsl.dataflow.memoization INFO: Task 3699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3699 try 0 launched on executor multi with executor id 3700 parsl.dataflow.dflow INFO: Standard output for task 3699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/75d13548-9d57-4bdf-845c-66dd323d00ba_isrForCrosstalkSources_1125906_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/75d13548-9d57-4bdf-845c-66dd323d00ba_isrForCrosstalkSources_1125906_41.stderr parsl.dataflow.dflow DEBUG: Task 3700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3700 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3700 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3700 has memoization hash 588be9808173f099005f8ace351397b4 parsl.dataflow.memoization INFO: Task 3700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3700 try 0 launched on executor multi with executor id 3701 parsl.dataflow.dflow INFO: Standard output for task 3700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/94f8d5d2-50f9-4f03-81c3-33cc1f122792_isrForCrosstalkSources_1125929_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/94f8d5d2-50f9-4f03-81c3-33cc1f122792_isrForCrosstalkSources_1125929_42.stderr parsl.dataflow.dflow DEBUG: Task 3701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3701 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3701 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3701 has memoization hash 6bcbf71a73020e7594c79c46c4297fdf parsl.dataflow.memoization INFO: Task 3701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3701 try 0 launched on executor multi with executor id 3702 parsl.dataflow.dflow INFO: Standard output for task 3701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/7de65aa6-1091-4f31-9728-e224424ca62a_isrForCrosstalkSources_1125905_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/7de65aa6-1091-4f31-9728-e224424ca62a_isrForCrosstalkSources_1125905_28.stderr parsl.dataflow.dflow DEBUG: Task 3702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3702 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3702 has memoization hash 8a89adc5f0d5c700be2b9c4f4e4e2016 parsl.dataflow.memoization INFO: Task 3702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3702 try 0 launched on executor multi with executor id 3703 parsl.dataflow.dflow INFO: Standard output for task 3702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/0c14f014-5abe-4adb-9230-6941dbec48f7_isrForCrosstalkSources_1125923_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/0c14f014-5abe-4adb-9230-6941dbec48f7_isrForCrosstalkSources_1125923_18.stderr parsl.dataflow.dflow DEBUG: Task 3703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3703 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3703 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3703 has memoization hash c80365a1891ebfe367d2aecf8b259c4c parsl.dataflow.memoization INFO: Task 3703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3703 try 0 launched on executor multi with executor id 3704 parsl.dataflow.dflow INFO: Standard output for task 3703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/19430e54-24ef-43e4-a008-150b1be860ad_isrForCrosstalkSources_1125926_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/19430e54-24ef-43e4-a008-150b1be860ad_isrForCrosstalkSources_1125926_1.stderr parsl.dataflow.dflow DEBUG: Task 3704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3704 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3704 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3704 has memoization hash 7df2c7e5926fa37be874a4118e730692 parsl.dataflow.memoization INFO: Task 3704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3704 try 0 launched on executor multi with executor id 3705 parsl.dataflow.dflow INFO: Standard output for task 3704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/375729e2-c542-41c5-bd64-04361b54919f_isrForCrosstalkSources_1125940_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/375729e2-c542-41c5-bd64-04361b54919f_isrForCrosstalkSources_1125940_23.stderr parsl.dataflow.dflow DEBUG: Task 3705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3705 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3705 has memoization hash 1258c336f28dfc35b535f5e50972771d parsl.dataflow.memoization INFO: Task 3705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3705 try 0 launched on executor multi with executor id 3706 parsl.dataflow.dflow INFO: Standard output for task 3705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/d590dcfa-68e7-46e6-a16e-d1cbfcb00873_isrForCrosstalkSources_1125928_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/d590dcfa-68e7-46e6-a16e-d1cbfcb00873_isrForCrosstalkSources_1125928_7.stderr parsl.dataflow.dflow DEBUG: Task 3706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3706 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3706 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3706 has memoization hash 176a3af87e09ec2c515b3a5b63b3039e parsl.dataflow.memoization INFO: Task 3706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3706 try 0 launched on executor multi with executor id 3707 parsl.dataflow.dflow INFO: Standard output for task 3706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/15cbd580-ce6e-43be-aa64-ccb7c343e1e6_isrForCrosstalkSources_1125935_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/15cbd580-ce6e-43be-aa64-ccb7c343e1e6_isrForCrosstalkSources_1125935_36.stderr parsl.dataflow.dflow DEBUG: Task 3707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3707 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3707 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3707 has memoization hash 86b54064ecc4242afff74b3461ad2de2 parsl.dataflow.memoization INFO: Task 3707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3707 try 0 launched on executor multi with executor id 3708 parsl.dataflow.dflow INFO: Standard output for task 3707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/b3fa840b-5926-4b81-9cb8-cfa197cd43ab_isrForCrosstalkSources_1125922_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/b3fa840b-5926-4b81-9cb8-cfa197cd43ab_isrForCrosstalkSources_1125922_1.stderr parsl.dataflow.dflow DEBUG: Task 3708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3708 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3708 has memoization hash 8de949b43ac16a8dfb722ebc4a624f74 parsl.dataflow.memoization INFO: Task 3708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3708 try 0 launched on executor multi with executor id 3709 parsl.dataflow.dflow INFO: Standard output for task 3708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/90895def-532b-4031-a2df-f65562341e35_isrForCrosstalkSources_1125922_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/90895def-532b-4031-a2df-f65562341e35_isrForCrosstalkSources_1125922_21.stderr parsl.dataflow.dflow DEBUG: Task 3709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3709 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3709 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3709 has memoization hash 5088e905601e3dfcf29c6dff1817e8e0 parsl.dataflow.memoization INFO: Task 3709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3709 try 0 launched on executor multi with executor id 3710 parsl.dataflow.dflow INFO: Standard output for task 3709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/2fe0c35e-202b-47f1-bbd2-a1f1f5f9a04f_isrForCrosstalkSources_1125932_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/2fe0c35e-202b-47f1-bbd2-a1f1f5f9a04f_isrForCrosstalkSources_1125932_13.stderr parsl.dataflow.dflow DEBUG: Task 3710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3710 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3710 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3710 has memoization hash 89a43428e4ff769232c8e3b904abf90a parsl.dataflow.memoization INFO: Task 3710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3710 try 0 launched on executor multi with executor id 3711 parsl.dataflow.dflow INFO: Standard output for task 3710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/45af3afb-6af1-44c7-9f19-145f5aead9a2_isrForCrosstalkSources_1125925_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/45af3afb-6af1-44c7-9f19-145f5aead9a2_isrForCrosstalkSources_1125925_1.stderr parsl.dataflow.dflow DEBUG: Task 3711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3711 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3711 has memoization hash e3be88c84fb20087bcecc084f1d03d49 parsl.dataflow.memoization INFO: Task 3711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3711 try 0 launched on executor multi with executor id 3712 parsl.dataflow.dflow INFO: Standard output for task 3711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/692452d7-c829-4a29-a636-1262a9baae8e_isrForCrosstalkSources_1125916_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/692452d7-c829-4a29-a636-1262a9baae8e_isrForCrosstalkSources_1125916_25.stderr parsl.dataflow.dflow DEBUG: Task 3712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3712 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3712 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3712 has memoization hash 69662969415c9fd6a226d2e20b3b4b2d parsl.dataflow.memoization INFO: Task 3712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3712 try 0 launched on executor multi with executor id 3713 parsl.dataflow.dflow INFO: Standard output for task 3712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/76cd4f4e-ac34-4988-9eb7-eae319839394_isrForCrosstalkSources_1125902_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/76cd4f4e-ac34-4988-9eb7-eae319839394_isrForCrosstalkSources_1125902_7.stderr parsl.dataflow.dflow DEBUG: Task 3713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3713 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3713 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3713 has memoization hash 180300531d60ada59f456d91a8818920 parsl.dataflow.memoization INFO: Task 3713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3713 try 0 launched on executor multi with executor id 3714 parsl.dataflow.dflow INFO: Standard output for task 3713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/36794827-8277-499d-a62f-cc5872d0f2fb_isrForCrosstalkSources_1125911_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/36794827-8277-499d-a62f-cc5872d0f2fb_isrForCrosstalkSources_1125911_26.stderr parsl.dataflow.dflow DEBUG: Task 3714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3714 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3714 has memoization hash b59401d3c6c9e697e42ea557f179a91e parsl.dataflow.memoization INFO: Task 3714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3714 try 0 launched on executor multi with executor id 3715 parsl.dataflow.dflow INFO: Standard output for task 3714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/6f19b8b8-2d4b-47e6-b2a1-0ed0beb500a7_isrForCrosstalkSources_1125902_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/6f19b8b8-2d4b-47e6-b2a1-0ed0beb500a7_isrForCrosstalkSources_1125902_30.stderr parsl.dataflow.dflow DEBUG: Task 3715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3715 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3715 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3715 has memoization hash 577182a2a8355a098bb9578e45ce74b7 parsl.dataflow.memoization INFO: Task 3715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3715 try 0 launched on executor multi with executor id 3716 parsl.dataflow.dflow INFO: Standard output for task 3715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/0731bc6b-98ce-4004-adec-0aec5594ffd2_isrForCrosstalkSources_1125922_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/0731bc6b-98ce-4004-adec-0aec5594ffd2_isrForCrosstalkSources_1125922_5.stderr parsl.dataflow.dflow DEBUG: Task 3716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3716 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3716 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3716 has memoization hash c8c64ae36d3ef0db16c33adef7dabba3 parsl.dataflow.memoization INFO: Task 3716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3716 try 0 launched on executor multi with executor id 3717 parsl.dataflow.dflow INFO: Standard output for task 3716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/d1ea00f0-054a-46bb-aaf2-67aa09b1a9ea_isrForCrosstalkSources_1125926_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/d1ea00f0-054a-46bb-aaf2-67aa09b1a9ea_isrForCrosstalkSources_1125926_8.stderr parsl.dataflow.dflow DEBUG: Task 3717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3717 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3717 has memoization hash 6595ce397d5fd63af2b943f1c2a145c6 parsl.dataflow.memoization INFO: Task 3717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3717 try 0 launched on executor multi with executor id 3718 parsl.dataflow.dflow INFO: Standard output for task 3717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/be3a84f5-a8f8-4dba-ab8e-effe19f171f8_isrForCrosstalkSources_1125891_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/be3a84f5-a8f8-4dba-ab8e-effe19f171f8_isrForCrosstalkSources_1125891_18.stderr parsl.dataflow.dflow DEBUG: Task 3718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3718 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3718 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3718 has memoization hash 66848c9383fc38615bd41294865d4388 parsl.dataflow.memoization INFO: Task 3718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3718 try 0 launched on executor multi with executor id 3719 parsl.dataflow.dflow INFO: Standard output for task 3718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/5d47d6df-d318-43d2-aaad-64a3cbb96151_isrForCrosstalkSources_1125931_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/5d47d6df-d318-43d2-aaad-64a3cbb96151_isrForCrosstalkSources_1125931_35.stderr parsl.dataflow.dflow DEBUG: Task 3719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3719 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3719 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3719 has memoization hash 9a317561ffa094a1e2993393108ee1eb parsl.dataflow.memoization INFO: Task 3719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x145a38f4d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 15:07:11 proc_lsst.multi:170 [INFO] [multi] job_id 16896454 proc_lsst.multi INFO: [multi] job_id 16896454 2024-03-07 15:07:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 15:07:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896454 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896454 parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 16896454 parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-07 15:07:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-07 15:07:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 15:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 3719 try 0 launched on executor multi with executor id 3720 parsl.dataflow.dflow INFO: Standard output for task 3719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/ee8fdaa0-0d94-4e3f-83ae-bb19f962adc7_isrForCrosstalkSources_1125891_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/ee8fdaa0-0d94-4e3f-83ae-bb19f962adc7_isrForCrosstalkSources_1125891_6.stderr parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:11 proc_lsst.multi:170 [INFO] [multi] job_id 16896455 proc_lsst.multi INFO: [multi] job_id 16896455 2024-03-07 15:07:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 15:07:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896455 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896455 parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 16896455 parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-07 15:07:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-07 15:07:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 15:07:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:12 proc_lsst.multi:170 [INFO] [multi] job_id 16896456 proc_lsst.multi INFO: [multi] job_id 16896456 2024-03-07 15:07:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 15:07:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896456 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896456 parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 16896456 parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-07 15:07:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:12 proc_lsst.multi:170 [INFO] [multi] job_id 16896457 proc_lsst.multi INFO: [multi] job_id 16896457 2024-03-07 15:07:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 15:07:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896457 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896457 parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 16896457 parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-07 15:07:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b3e44efc258b241091ff2c564306423 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/4b53e8d4-07b0-4089-b25e-5f5b20cc3ba8_isrForCrosstalkSources_1125924_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/4b53e8d4-07b0-4089-b25e-5f5b20cc3ba8_isrForCrosstalkSources_1125924_49.stderr 2024-03-07 15:07:12 proc_lsst.multi:170 [INFO] [multi] job_id 16896458 proc_lsst.multi INFO: [multi] job_id 16896458 2024-03-07 15:07:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 15:07:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896458 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896458 parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 16896458 parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-07 15:07:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:12 proc_lsst.multi:170 [INFO] [multi] job_id 16896459 proc_lsst.multi INFO: [multi] job_id 16896459 2024-03-07 15:07:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 15:07:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896459 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896459 parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 16896459 parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-07 15:07:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:12 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 15:07:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:13 proc_lsst.multi:170 [INFO] [multi] job_id 16896460 proc_lsst.multi INFO: [multi] job_id 16896460 2024-03-07 15:07:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 15:07:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:13 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896460 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896460 parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 16896460 parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-07 15:07:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-07 15:07:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:13 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 15:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:13 proc_lsst.multi:170 [INFO] [multi] job_id 16896461 proc_lsst.multi INFO: [multi] job_id 16896461 2024-03-07 15:07:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 15:07:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:13 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896461 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896461 parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 16896461 parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-07 15:07:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-07 15:07:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:13 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 15:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:13 proc_lsst.multi:170 [INFO] [multi] job_id 16896462 proc_lsst.multi INFO: [multi] job_id 16896462 2024-03-07 15:07:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 15:07:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:13 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896462 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896462 parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 16896462 parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-07 15:07:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-07 15:07:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:13 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 15:07:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:14 proc_lsst.multi:170 [INFO] [multi] job_id 16896463 proc_lsst.multi INFO: [multi] job_id 16896463 2024-03-07 15:07:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 15:07:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896463 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896463 parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 16896463 parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-07 15:07:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-07 15:07:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 15:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:14 proc_lsst.multi:170 [INFO] [multi] job_id 16896464 proc_lsst.multi INFO: [multi] job_id 16896464 2024-03-07 15:07:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896464 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896464 parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 16896464 parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-07 15:07:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-07 15:07:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:14 proc_lsst.multi:170 [INFO] [multi] job_id 16896466 proc_lsst.multi INFO: [multi] job_id 16896466 2024-03-07 15:07:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 15:07:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896466 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896466 parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 16896466 parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-07 15:07:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-07 15:07:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 15:07:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:15 proc_lsst.multi:170 [INFO] [multi] job_id 16896467 proc_lsst.multi INFO: [multi] job_id 16896467 2024-03-07 15:07:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 15:07:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896467 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896467 parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 16896467 parsl.executors.status_handling INFO: Allocated block ID 78 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78 2024-03-07 15:07:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 2024-03-07 15:07:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 15:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:15 proc_lsst.multi:170 [INFO] [multi] job_id 16896468 proc_lsst.multi INFO: [multi] job_id 16896468 2024-03-07 15:07:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 15:07:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896468 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896468 parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 16896468 parsl.executors.status_handling INFO: Allocated block ID 79 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79 2024-03-07 15:07:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 2024-03-07 15:07:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 15:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:15 proc_lsst.multi:170 [INFO] [multi] job_id 16896469 proc_lsst.multi INFO: [multi] job_id 16896469 2024-03-07 15:07:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 15:07:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896469 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896469 parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 16896469 parsl.executors.status_handling INFO: Allocated block ID 80 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80 2024-03-07 15:07:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 2024-03-07 15:07:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 15:07:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:16 proc_lsst.multi:170 [INFO] [multi] job_id 16896470 proc_lsst.multi INFO: [multi] job_id 16896470 2024-03-07 15:07:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 15:07:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896470 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896470 parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 16896470 parsl.executors.status_handling INFO: Allocated block ID 81 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81 2024-03-07 15:07:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:16 proc_lsst.multi:170 [INFO] [multi] job_id 16896471 proc_lsst.multi INFO: [multi] job_id 16896471 2024-03-07 15:07:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 15:07:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896471 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896471 parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 16896471 parsl.executors.status_handling INFO: Allocated block ID 82 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82 2024-03-07 15:07:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:16 proc_lsst.multi:170 [INFO] [multi] job_id 16896472 proc_lsst.multi INFO: [multi] job_id 16896472 2024-03-07 15:07:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 15:07:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896472 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896472 parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 16896472 parsl.executors.status_handling INFO: Allocated block ID 83 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83 2024-03-07 15:07:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:16 proc_lsst.multi:170 [INFO] [multi] job_id 16896473 proc_lsst.multi INFO: [multi] job_id 16896473 2024-03-07 15:07:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 15:07:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896473 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896473 parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 16896473 parsl.executors.status_handling INFO: Allocated block ID 84 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84 2024-03-07 15:07:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:16 proc_lsst.multi:170 [INFO] [multi] job_id 16896474 proc_lsst.multi INFO: [multi] job_id 16896474 2024-03-07 15:07:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 15:07:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896474 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896474 parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 16896474 parsl.executors.status_handling INFO: Allocated block ID 85 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85 2024-03-07 15:07:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:16 proc_lsst.multi:170 [INFO] [multi] job_id 16896475 proc_lsst.multi INFO: [multi] job_id 16896475 2024-03-07 15:07:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896475 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896475 parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 16896475 parsl.executors.status_handling INFO: Allocated block ID 86 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86 2024-03-07 15:07:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:16 proc_lsst.multi:170 [INFO] [multi] job_id 16896476 proc_lsst.multi INFO: [multi] job_id 16896476 2024-03-07 15:07:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-07 15:07:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896476 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896476 parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 16896476 parsl.executors.status_handling INFO: Allocated block ID 87 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87 2024-03-07 15:07:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-07 15:07:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:17 proc_lsst.multi:170 [INFO] [multi] job_id 16896477 proc_lsst.multi INFO: [multi] job_id 16896477 2024-03-07 15:07:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-07 15:07:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896477 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896477 parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 16896477 parsl.executors.status_handling INFO: Allocated block ID 88 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88 2024-03-07 15:07:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:17 proc_lsst.multi:170 [INFO] [multi] job_id 16896478 proc_lsst.multi INFO: [multi] job_id 16896478 2024-03-07 15:07:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-07 15:07:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896478 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896478 parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 16896478 parsl.executors.status_handling INFO: Allocated block ID 89 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89 2024-03-07 15:07:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:17 proc_lsst.multi:170 [INFO] [multi] job_id 16896479 proc_lsst.multi INFO: [multi] job_id 16896479 2024-03-07 15:07:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-07 15:07:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896479 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896479 parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 16896479 parsl.executors.status_handling INFO: Allocated block ID 90 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90 2024-03-07 15:07:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:17 proc_lsst.multi:170 [INFO] [multi] job_id 16896480 proc_lsst.multi INFO: [multi] job_id 16896480 2024-03-07 15:07:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-07 15:07:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896480 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896480 parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 16896480 parsl.executors.status_handling INFO: Allocated block ID 91 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91 2024-03-07 15:07:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:17 proc_lsst.multi:170 [INFO] [multi] job_id 16896481 proc_lsst.multi INFO: [multi] job_id 16896481 2024-03-07 15:07:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-07 15:07:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896481 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896481 parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 16896481 parsl.executors.status_handling INFO: Allocated block ID 92 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92 2024-03-07 15:07:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:17 proc_lsst.multi:170 [INFO] [multi] job_id 16896482 proc_lsst.multi INFO: [multi] job_id 16896482 2024-03-07 15:07:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-07 15:07:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896482 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896482 parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 16896482 parsl.executors.status_handling INFO: Allocated block ID 93 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-93 2024-03-07 15:07:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-93 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-93 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-07 15:07:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:18 proc_lsst.multi:170 [INFO] [multi] job_id 16896483 proc_lsst.multi INFO: [multi] job_id 16896483 2024-03-07 15:07:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 38 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 38 2024-03-07 15:07:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896483 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896483 parsl.executors.status_handling DEBUG: Launched block 93 on executor multi with job ID 16896483 parsl.executors.status_handling INFO: Allocated block ID 94 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-94 2024-03-07 15:07:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-94 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-94 2024-03-07 15:07:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 38 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 38 2024-03-07 15:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:18 proc_lsst.multi:170 [INFO] [multi] job_id 16896484 proc_lsst.multi INFO: [multi] job_id 16896484 2024-03-07 15:07:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 39 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 39 2024-03-07 15:07:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896484 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896484 parsl.executors.status_handling DEBUG: Launched block 94 on executor multi with job ID 16896484 parsl.executors.status_handling INFO: Allocated block ID 95 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-95 2024-03-07 15:07:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-95 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-95 2024-03-07 15:07:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 39 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 39 2024-03-07 15:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:18 proc_lsst.multi:170 [INFO] [multi] job_id 16896485 proc_lsst.multi INFO: [multi] job_id 16896485 2024-03-07 15:07:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 40 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 40 2024-03-07 15:07:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896485 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896485 parsl.executors.status_handling DEBUG: Launched block 95 on executor multi with job ID 16896485 parsl.executors.status_handling INFO: Allocated block ID 96 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-96 2024-03-07 15:07:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-96 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-96 2024-03-07 15:07:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 40 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 40 2024-03-07 15:07:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:19 proc_lsst.multi:170 [INFO] [multi] job_id 16896486 proc_lsst.multi INFO: [multi] job_id 16896486 2024-03-07 15:07:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 41 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 41 2024-03-07 15:07:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896486 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896486 parsl.executors.status_handling DEBUG: Launched block 96 on executor multi with job ID 16896486 parsl.executors.status_handling INFO: Allocated block ID 97 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-97 2024-03-07 15:07:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-97 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-97 2024-03-07 15:07:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 41 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 41 2024-03-07 15:07:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:19 proc_lsst.multi:170 [INFO] [multi] job_id 16896487 proc_lsst.multi INFO: [multi] job_id 16896487 2024-03-07 15:07:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 42 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 42 2024-03-07 15:07:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896487 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896487 parsl.executors.status_handling DEBUG: Launched block 97 on executor multi with job ID 16896487 parsl.executors.status_handling INFO: Allocated block ID 98 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-98 2024-03-07 15:07:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-98 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-98 2024-03-07 15:07:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 42 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 42 2024-03-07 15:07:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:19 proc_lsst.multi:170 [INFO] [multi] job_id 16896488 proc_lsst.multi INFO: [multi] job_id 16896488 2024-03-07 15:07:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 43 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 43 2024-03-07 15:07:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896488 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896488 parsl.executors.status_handling DEBUG: Launched block 98 on executor multi with job ID 16896488 parsl.executors.status_handling INFO: Allocated block ID 99 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-99 2024-03-07 15:07:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-99 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-99 2024-03-07 15:07:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 43 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 43 2024-03-07 15:07:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:20 proc_lsst.multi:170 [INFO] [multi] job_id 16896489 proc_lsst.multi INFO: [multi] job_id 16896489 2024-03-07 15:07:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 44 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 44 2024-03-07 15:07:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896489 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896489 parsl.executors.status_handling DEBUG: Launched block 99 on executor multi with job ID 16896489 parsl.executors.status_handling INFO: Allocated block ID 100 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-100 2024-03-07 15:07:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-100 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-100 2024-03-07 15:07:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 44 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 44 2024-03-07 15:07:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:20 proc_lsst.multi:170 [INFO] [multi] job_id 16896490 proc_lsst.multi INFO: [multi] job_id 16896490 2024-03-07 15:07:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 45 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 45 2024-03-07 15:07:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896490 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896490 parsl.executors.status_handling DEBUG: Launched block 100 on executor multi with job ID 16896490 parsl.executors.status_handling INFO: Allocated block ID 101 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-101 2024-03-07 15:07:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-101 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-101 2024-03-07 15:07:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 45 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 45 2024-03-07 15:07:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:21 proc_lsst.multi:170 [INFO] [multi] job_id 16896491 proc_lsst.multi INFO: [multi] job_id 16896491 2024-03-07 15:07:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 46 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 46 2024-03-07 15:07:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896491 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896491 parsl.executors.status_handling DEBUG: Launched block 101 on executor multi with job ID 16896491 parsl.executors.status_handling INFO: Allocated block ID 102 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-102 2024-03-07 15:07:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-102 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-102 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 46 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 46 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:21 proc_lsst.multi:170 [INFO] [multi] job_id 16896492 proc_lsst.multi INFO: [multi] job_id 16896492 2024-03-07 15:07:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 47 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 47 2024-03-07 15:07:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896492 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896492 parsl.executors.status_handling DEBUG: Launched block 102 on executor multi with job ID 16896492 parsl.executors.status_handling INFO: Allocated block ID 103 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-103 2024-03-07 15:07:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-103 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-103 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 47 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 47 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:21 proc_lsst.multi:170 [INFO] [multi] job_id 16896493 proc_lsst.multi INFO: [multi] job_id 16896493 2024-03-07 15:07:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 48 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 48 2024-03-07 15:07:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896493 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896493 parsl.executors.status_handling DEBUG: Launched block 103 on executor multi with job ID 16896493 parsl.executors.status_handling INFO: Allocated block ID 104 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-104 2024-03-07 15:07:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-104 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-104 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 48 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 48 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:21 proc_lsst.multi:170 [INFO] [multi] job_id 16896494 proc_lsst.multi INFO: [multi] job_id 16896494 2024-03-07 15:07:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 49 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 49 2024-03-07 15:07:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896494 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896494 parsl.executors.status_handling DEBUG: Launched block 104 on executor multi with job ID 16896494 parsl.executors.status_handling INFO: Allocated block ID 105 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-105 2024-03-07 15:07:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-105 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-105 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 49 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 49 2024-03-07 15:07:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e7b6b36150b8a44f23c38b478d53f21 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/34cfcaa4-0aec-489d-90bf-b19fb701bb22_isrForCrosstalkSources_1125907_3.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/34cfcaa4-0aec-489d-90bf-b19fb701bb22_isrForCrosstalkSources_1125907_3.stderr 2024-03-07 15:07:22 proc_lsst.multi:170 [INFO] [multi] job_id 16896495 proc_lsst.multi INFO: [multi] job_id 16896495 2024-03-07 15:07:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 50 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 50 2024-03-07 15:07:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896495 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896495 parsl.executors.status_handling DEBUG: Launched block 105 on executor multi with job ID 16896495 parsl.executors.status_handling INFO: Allocated block ID 106 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-106 2024-03-07 15:07:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-106 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-106 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 50 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 50 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:22 proc_lsst.multi:170 [INFO] [multi] job_id 16896496 proc_lsst.multi INFO: [multi] job_id 16896496 2024-03-07 15:07:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 51 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 51 2024-03-07 15:07:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896496 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896496 parsl.executors.status_handling DEBUG: Launched block 106 on executor multi with job ID 16896496 parsl.executors.status_handling INFO: Allocated block ID 107 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-107 2024-03-07 15:07:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-107 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-107 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 51 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 51 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 430f82ed7633812ead111e1dff08aa49 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/5fe4ba29-d820-48ea-992d-0abc622a58b1_isrForCrosstalkSources_1125932_28.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/5fe4ba29-d820-48ea-992d-0abc622a58b1_isrForCrosstalkSources_1125932_28.stderr 2024-03-07 15:07:22 proc_lsst.multi:170 [INFO] [multi] job_id 16896497 proc_lsst.multi INFO: [multi] job_id 16896497 2024-03-07 15:07:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 52 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 52 2024-03-07 15:07:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896497 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896497 parsl.executors.status_handling DEBUG: Launched block 107 on executor multi with job ID 16896497 parsl.executors.status_handling INFO: Allocated block ID 108 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-108 2024-03-07 15:07:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-108 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-108 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 52 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 52 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:22 proc_lsst.multi:170 [INFO] [multi] job_id 16896498 proc_lsst.multi INFO: [multi] job_id 16896498 2024-03-07 15:07:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 53 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 53 2024-03-07 15:07:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896498 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896498 parsl.executors.status_handling DEBUG: Launched block 108 on executor multi with job ID 16896498 parsl.executors.status_handling INFO: Allocated block ID 109 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-109 2024-03-07 15:07:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-109 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-109 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 53 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 53 2024-03-07 15:07:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:23 proc_lsst.multi:170 [INFO] [multi] job_id 16896499 proc_lsst.multi INFO: [multi] job_id 16896499 2024-03-07 15:07:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 54 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 54 2024-03-07 15:07:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896499 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896499 parsl.executors.status_handling DEBUG: Launched block 109 on executor multi with job ID 16896499 parsl.executors.status_handling INFO: Allocated block ID 110 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-110 2024-03-07 15:07:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-110 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-110 2024-03-07 15:07:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 54 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 54 2024-03-07 15:07:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:23 proc_lsst.multi:170 [INFO] [multi] job_id 16896500 proc_lsst.multi INFO: [multi] job_id 16896500 2024-03-07 15:07:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 55 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 55 2024-03-07 15:07:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896500 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896500 parsl.executors.status_handling DEBUG: Launched block 110 on executor multi with job ID 16896500 parsl.executors.status_handling INFO: Allocated block ID 111 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-111 2024-03-07 15:07:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-111 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-111 2024-03-07 15:07:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 55 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 55 2024-03-07 15:07:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:24 proc_lsst.multi:170 [INFO] [multi] job_id 16896501 proc_lsst.multi INFO: [multi] job_id 16896501 2024-03-07 15:07:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 56 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 56 2024-03-07 15:07:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896501 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896501 parsl.executors.status_handling DEBUG: Launched block 111 on executor multi with job ID 16896501 parsl.executors.status_handling INFO: Allocated block ID 112 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-112 2024-03-07 15:07:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-112 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-112 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 56 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 56 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:24 proc_lsst.multi:170 [INFO] [multi] job_id 16896502 proc_lsst.multi INFO: [multi] job_id 16896502 2024-03-07 15:07:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 57 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 57 2024-03-07 15:07:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896502 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896502 parsl.executors.status_handling DEBUG: Launched block 112 on executor multi with job ID 16896502 parsl.executors.status_handling INFO: Allocated block ID 113 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-113 2024-03-07 15:07:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-113 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-113 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 57 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 57 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:24 proc_lsst.multi:170 [INFO] [multi] job_id 16896503 proc_lsst.multi INFO: [multi] job_id 16896503 2024-03-07 15:07:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 58 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 58 2024-03-07 15:07:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896503 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896503 parsl.executors.status_handling DEBUG: Launched block 113 on executor multi with job ID 16896503 parsl.executors.status_handling INFO: Allocated block ID 114 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-114 2024-03-07 15:07:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-114 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-114 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 58 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 58 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:24 proc_lsst.multi:170 [INFO] [multi] job_id 16896504 proc_lsst.multi INFO: [multi] job_id 16896504 2024-03-07 15:07:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 59 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 59 2024-03-07 15:07:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896504 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896504 parsl.executors.status_handling DEBUG: Launched block 114 on executor multi with job ID 16896504 parsl.executors.status_handling INFO: Allocated block ID 115 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-115 2024-03-07 15:07:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-115 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-115 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 59 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 59 2024-03-07 15:07:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:25 proc_lsst.multi:170 [INFO] [multi] job_id 16896505 proc_lsst.multi INFO: [multi] job_id 16896505 2024-03-07 15:07:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 60 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 60 2024-03-07 15:07:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896505 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896505 parsl.executors.status_handling DEBUG: Launched block 115 on executor multi with job ID 16896505 parsl.executors.status_handling INFO: Allocated block ID 116 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-116 2024-03-07 15:07:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-116 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-116 2024-03-07 15:07:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 60 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 60 2024-03-07 15:07:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:25 proc_lsst.multi:170 [INFO] [multi] job_id 16896506 proc_lsst.multi INFO: [multi] job_id 16896506 2024-03-07 15:07:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 61 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 61 2024-03-07 15:07:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896506 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896506 parsl.executors.status_handling DEBUG: Launched block 116 on executor multi with job ID 16896506 parsl.executors.status_handling INFO: Allocated block ID 117 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-117 2024-03-07 15:07:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-117 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-117 2024-03-07 15:07:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 61 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 61 2024-03-07 15:07:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:26 proc_lsst.multi:170 [INFO] [multi] job_id 16896507 proc_lsst.multi INFO: [multi] job_id 16896507 2024-03-07 15:07:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 62 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 62 2024-03-07 15:07:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896507 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896507 parsl.executors.status_handling DEBUG: Launched block 117 on executor multi with job ID 16896507 parsl.executors.status_handling INFO: Allocated block ID 118 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-118 2024-03-07 15:07:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-118 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-118 2024-03-07 15:07:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 62 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 62 2024-03-07 15:07:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:26 proc_lsst.multi:170 [INFO] [multi] job_id 16896508 proc_lsst.multi INFO: [multi] job_id 16896508 2024-03-07 15:07:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 63 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 63 2024-03-07 15:07:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896508 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896508 parsl.executors.status_handling DEBUG: Launched block 118 on executor multi with job ID 16896508 parsl.executors.status_handling INFO: Allocated block ID 119 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-119 2024-03-07 15:07:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-119 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-119 2024-03-07 15:07:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 63 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 63 2024-03-07 15:07:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:28 proc_lsst.multi:170 [INFO] [multi] job_id 16896509 proc_lsst.multi INFO: [multi] job_id 16896509 2024-03-07 15:07:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 64 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 64 2024-03-07 15:07:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896509 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896509 parsl.executors.status_handling DEBUG: Launched block 119 on executor multi with job ID 16896509 parsl.executors.status_handling INFO: Allocated block ID 120 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-120 2024-03-07 15:07:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-120 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-120 2024-03-07 15:07:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 64 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 64 2024-03-07 15:07:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 356d301980b39f5470c23232c0b4f321 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/db1cc29b-62d3-4e77-86e6-31a450abcc15_isrForCrosstalkSources_1125923_56.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/db1cc29b-62d3-4e77-86e6-31a450abcc15_isrForCrosstalkSources_1125923_56.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92352105e1a9248e5e774eda1a6f0059 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/150d1312-4cab-4e5e-9488-15f6f7400d19_isrForCrosstalkSources_1125906_37.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/150d1312-4cab-4e5e-9488-15f6f7400d19_isrForCrosstalkSources_1125906_37.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c3e6568de0ed0e3be1dd58eb1a99566 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/2ad9bc49-1f7d-44f6-b7e8-37759a799852_isrForCrosstalkSources_1125919_48.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/2ad9bc49-1f7d-44f6-b7e8-37759a799852_isrForCrosstalkSources_1125919_48.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 848ee740662519e9ba15522b08e9176c with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/6b34c6a4-a11e-4ffe-b729-2061134d1ffd_isrForCrosstalkSources_1125905_13.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/6b34c6a4-a11e-4ffe-b729-2061134d1ffd_isrForCrosstalkSources_1125905_13.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a65444b890817ca6b8ae5ac7febf87b1 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/c713b05f-0d83-4d4b-a5cb-89b74fdd358c_isrForCrosstalkSources_1125942_59.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/c713b05f-0d83-4d4b-a5cb-89b74fdd358c_isrForCrosstalkSources_1125942_59.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6efcba1d56ac6afb008ef242621296e3 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/a92ef0be-cc6b-42a9-b761-d9ea513e4a9f_isrForCrosstalkSources_1125919_60.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/a92ef0be-cc6b-42a9-b761-d9ea513e4a9f_isrForCrosstalkSources_1125919_60.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efe1a61d39a5b2f523c760a1394e6899 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/ecaaf644-5219-4376-bf6f-d4c500186426_isrForCrosstalkSources_1125944_7.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/ecaaf644-5219-4376-bf6f-d4c500186426_isrForCrosstalkSources_1125944_7.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4529941d2b71479badae19dd2459f9f8 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/810670c6-f9ae-49a7-95b4-245f9a2bf2b8_isrForCrosstalkSources_1125943_36.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/810670c6-f9ae-49a7-95b4-245f9a2bf2b8_isrForCrosstalkSources_1125943_36.stderr 2024-03-07 15:07:28 proc_lsst.multi:170 [INFO] [multi] job_id 16896510 proc_lsst.multi INFO: [multi] job_id 16896510 2024-03-07 15:07:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 65 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 65 2024-03-07 15:07:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896510 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896510 parsl.executors.status_handling DEBUG: Launched block 120 on executor multi with job ID 16896510 parsl.executors.status_handling INFO: Allocated block ID 121 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-121 2024-03-07 15:07:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-121 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-121 2024-03-07 15:07:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:07:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:07:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:07:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:07:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:07:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:07:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:07:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:07:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:07:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:07:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 65 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 65 2024-03-07 15:07:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:07:29 proc_lsst.multi:170 [INFO] [multi] job_id 16896511 proc_lsst.multi INFO: [multi] job_id 16896511 2024-03-07 15:07:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 66 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 66 2024-03-07 15:07:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:07:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896511 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896511 parsl.executors.status_handling DEBUG: Launched block 121 on executor multi with job ID 16896511 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22377792571472 2024-03-07 15:07:29 proc_lsst.multi:146 [INFO] found job 53087 in provider local proc_lsst.multi INFO: found job 53087 in provider local 2024-03-07 15:07:29 proc_lsst.multi:146 [INFO] found job 16896385 in provider astro proc_lsst.multi INFO: found job 16896385 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:29 proc_lsst.multi:146 [INFO] found job 16896386 in provider astro proc_lsst.multi INFO: found job 16896386 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:30 proc_lsst.multi:146 [INFO] found job 16896387 in provider astro proc_lsst.multi INFO: found job 16896387 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:30 proc_lsst.multi:146 [INFO] found job 16896388 in provider astro proc_lsst.multi INFO: found job 16896388 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:31 proc_lsst.multi:146 [INFO] found job 16896389 in provider astro proc_lsst.multi INFO: found job 16896389 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba7701fa01266c1e84f4caca359f51d5 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/ca5e3bde-9058-47e9-8ece-b411e79500c2_isrForCrosstalkSources_1125937_51.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/ca5e3bde-9058-47e9-8ece-b411e79500c2_isrForCrosstalkSources_1125937_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:31 proc_lsst.multi:146 [INFO] found job 16896390 in provider astro proc_lsst.multi INFO: found job 16896390 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:32 proc_lsst.multi:146 [INFO] found job 16896392 in provider astro proc_lsst.multi INFO: found job 16896392 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5fa84d2e60658cc9635dc31f48a0422 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/69fb2474-85a2-46a8-aa15-3a68a0b5eae3_isrForCrosstalkSources_1125890_31.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/69fb2474-85a2-46a8-aa15-3a68a0b5eae3_isrForCrosstalkSources_1125890_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:32 proc_lsst.multi:146 [INFO] found job 16896394 in provider astro proc_lsst.multi INFO: found job 16896394 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:33 proc_lsst.multi:146 [INFO] found job 16896395 in provider astro proc_lsst.multi INFO: found job 16896395 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:33 proc_lsst.multi:146 [INFO] found job 16896396 in provider astro proc_lsst.multi INFO: found job 16896396 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:34 proc_lsst.multi:146 [INFO] found job 16896397 in provider astro proc_lsst.multi INFO: found job 16896397 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:34 proc_lsst.multi:146 [INFO] found job 16896398 in provider astro proc_lsst.multi INFO: found job 16896398 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:35 proc_lsst.multi:146 [INFO] found job 16896399 in provider astro proc_lsst.multi INFO: found job 16896399 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:35 proc_lsst.multi:146 [INFO] found job 16896400 in provider astro proc_lsst.multi INFO: found job 16896400 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:36 proc_lsst.multi:146 [INFO] found job 16896401 in provider astro proc_lsst.multi INFO: found job 16896401 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:36 proc_lsst.multi:146 [INFO] found job 16896402 in provider astro proc_lsst.multi INFO: found job 16896402 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:36 proc_lsst.multi:146 [INFO] found job 16896403 in provider astro proc_lsst.multi INFO: found job 16896403 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:36 proc_lsst.multi:146 [INFO] found job 16896404 in provider astro proc_lsst.multi INFO: found job 16896404 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:37 proc_lsst.multi:146 [INFO] found job 16896405 in provider astro proc_lsst.multi INFO: found job 16896405 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:37 proc_lsst.multi:146 [INFO] found job 16896408 in provider astro proc_lsst.multi INFO: found job 16896408 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:37 proc_lsst.multi:146 [INFO] found job 16896409 in provider astro proc_lsst.multi INFO: found job 16896409 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:37 proc_lsst.multi:146 [INFO] found job 16896410 in provider astro proc_lsst.multi INFO: found job 16896410 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:38 proc_lsst.multi:146 [INFO] found job 16896411 in provider astro proc_lsst.multi INFO: found job 16896411 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:38 proc_lsst.multi:146 [INFO] found job 16896412 in provider astro proc_lsst.multi INFO: found job 16896412 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90d99311e6aac81c2d55eba3143a6c35 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/18ff2f1b-7803-4649-8041-da62e43fcdf2_isrForCrosstalkSources_1125946_32.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/18ff2f1b-7803-4649-8041-da62e43fcdf2_isrForCrosstalkSources_1125946_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:39 proc_lsst.multi:146 [INFO] found job 16896413 in provider astro proc_lsst.multi INFO: found job 16896413 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b783485cf434f91fa89e4da8e01240a7 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/e89d2964-ad20-4784-abb9-821d847103ef_isrForCrosstalkSources_1125935_19.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/e89d2964-ad20-4784-abb9-821d847103ef_isrForCrosstalkSources_1125935_19.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53ca7f5c91a3b29132a49a72eeba15ae with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/c599ec5a-4a57-4daa-8707-30e623b4b2c2_isrForCrosstalkSources_1125936_10.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/c599ec5a-4a57-4daa-8707-30e623b4b2c2_isrForCrosstalkSources_1125936_10.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 495d1e9b1c86f23b6dd357cc80a6c21e with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/3e34db29-c1a9-4059-ba41-52f73b0d5d6d_isrForCrosstalkSources_1125931_10.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/3e34db29-c1a9-4059-ba41-52f73b0d5d6d_isrForCrosstalkSources_1125931_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:39 proc_lsst.multi:146 [INFO] found job 16896414 in provider astro proc_lsst.multi INFO: found job 16896414 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94e891b69ca653f3e2367e214b317e12 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f165198d-30b0-43cf-b992-2f7aec3cc189_isrForCrosstalkSources_1125922_18.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f165198d-30b0-43cf-b992-2f7aec3cc189_isrForCrosstalkSources_1125922_18.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cde9f318e6f8fbf7e82af92ddf26808c with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/b7faa7cc-e2a2-458e-a82b-64e5bf64b3af_isrForCrosstalkSources_1125942_53.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/b7faa7cc-e2a2-458e-a82b-64e5bf64b3af_isrForCrosstalkSources_1125942_53.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79646ae77039b8ead4df5c32b9fd57d8 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/ace97819-a2a7-4167-9284-2159de2654e5_isrForCrosstalkSources_1125940_29.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/ace97819-a2a7-4167-9284-2159de2654e5_isrForCrosstalkSources_1125940_29.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5d581aedccc44f630b08a567fa84e26 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/4a571317-7ff9-480e-b8c1-0d67fe467fb9_isrForCrosstalkSources_1125892_60.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/4a571317-7ff9-480e-b8c1-0d67fe467fb9_isrForCrosstalkSources_1125892_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:40 proc_lsst.multi:146 [INFO] found job 16896415 in provider astro proc_lsst.multi INFO: found job 16896415 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:40 proc_lsst.multi:146 [INFO] found job 16896416 in provider astro proc_lsst.multi INFO: found job 16896416 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:41 proc_lsst.multi:146 [INFO] found job 16896417 in provider astro proc_lsst.multi INFO: found job 16896417 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a59f96d69c03d285c8d0e51c77d42846 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/2a03a04f-3380-42f8-9145-31b6bb6ae884_isrForCrosstalkSources_1125934_1.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/2a03a04f-3380-42f8-9145-31b6bb6ae884_isrForCrosstalkSources_1125934_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:41 proc_lsst.multi:146 [INFO] found job 16896418 in provider astro proc_lsst.multi INFO: found job 16896418 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:41 proc_lsst.multi:146 [INFO] found job 16896419 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896419 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6135e3bf459376d6bd5d881658a32798 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/3bd8e76c-52c6-412f-9627-eff561f636c8_isrForCrosstalkSources_1125907_29.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/3bd8e76c-52c6-412f-9627-eff561f636c8_isrForCrosstalkSources_1125907_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:42 proc_lsst.multi:146 [INFO] found job 16896420 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896420 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:42 proc_lsst.multi:146 [INFO] found job 16896421 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896421 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:43 proc_lsst.multi:146 [INFO] found job 16896422 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896422 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:43 proc_lsst.multi:146 [INFO] found job 16896423 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896423 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:44 proc_lsst.multi:146 [INFO] found job 16896424 in provider escience_a40 proc_lsst.multi INFO: found job 16896424 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 365e35489285713e7878b7eed940caa5 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/30677ff4-043f-42db-a120-bb410320c937_isrForCrosstalkSources_1125936_18.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/30677ff4-043f-42db-a120-bb410320c937_isrForCrosstalkSources_1125936_18.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b629429c3ccec3b1db866e355f3524dd with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/30fcc946-ad00-4835-a36d-d1ae86bd7e6c_isrForCrosstalkSources_1125902_27.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/30fcc946-ad00-4835-a36d-d1ae86bd7e6c_isrForCrosstalkSources_1125902_27.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7090b80dd3788c5e73bdb583a05bacf9 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/efd30137-cc45-4c48-8afe-0cdf27209dae_isrForCrosstalkSources_1125903_19.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/efd30137-cc45-4c48-8afe-0cdf27209dae_isrForCrosstalkSources_1125903_19.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70738b2f0c86c7090bcf37b803c42218 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/067690b7-318a-4088-b270-75459da3dbe9_isrForCrosstalkSources_1125923_55.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/067690b7-318a-4088-b270-75459da3dbe9_isrForCrosstalkSources_1125923_55.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdef1aa1f00206cc6997d5ce20b4ceb7 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9c397d48-9c95-4026-bbe8-8064f1113ed3_isrForCrosstalkSources_1125921_11.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9c397d48-9c95-4026-bbe8-8064f1113ed3_isrForCrosstalkSources_1125921_11.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ab99a83aeef6da1ad79994519635499 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0c3fe679-7751-490b-89c5-5e5fcffc77d2_isrForCrosstalkSources_1125890_24.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0c3fe679-7751-490b-89c5-5e5fcffc77d2_isrForCrosstalkSources_1125890_24.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee5b06594ba2cb68b64f014320058be8 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/7cdcc8ea-6810-42c8-b0d3-0b0a591629ba_isrForCrosstalkSources_1125895_22.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/7cdcc8ea-6810-42c8-b0d3-0b0a591629ba_isrForCrosstalkSources_1125895_22.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03479548d81a516c847ae4bf05a51380 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/93728636-a3ac-4b29-8671-0b935c0a86af_isrForCrosstalkSources_1125931_56.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/93728636-a3ac-4b29-8671-0b935c0a86af_isrForCrosstalkSources_1125931_56.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50014a4ef377ba9055011635267a7c21 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/9ca3a0fd-42f4-49c3-b6d3-97505f29b1e5_isrForCrosstalkSources_1125898_57.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/9ca3a0fd-42f4-49c3-b6d3-97505f29b1e5_isrForCrosstalkSources_1125898_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:44 proc_lsst.multi:146 [INFO] found job 16896425 in provider escience_a40 proc_lsst.multi INFO: found job 16896425 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41998944fbbac9c82d0a1551fc8dcde4 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f36a0baf-93ff-43ac-ac7b-18d6901c828b_isrForCrosstalkSources_1125897_48.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f36a0baf-93ff-43ac-ac7b-18d6901c828b_isrForCrosstalkSources_1125897_48.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1109c5f0852bfdf7ea4c097c55d8a464 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/015d8094-bf7d-4078-9674-9911c5415c67_isrForCrosstalkSources_1125919_62.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/015d8094-bf7d-4078-9674-9911c5415c67_isrForCrosstalkSources_1125919_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:45 proc_lsst.multi:146 [INFO] found job 16896426 in provider escience_a40 proc_lsst.multi INFO: found job 16896426 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:45 proc_lsst.multi:146 [INFO] found job 16896427 in provider escience_a40 proc_lsst.multi INFO: found job 16896427 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:46 proc_lsst.multi:146 [INFO] found job 16896428 in provider escience_a40 proc_lsst.multi INFO: found job 16896428 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:46 proc_lsst.multi:146 [INFO] found job 16896429 in provider escience_a40 proc_lsst.multi INFO: found job 16896429 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2976fabed2c6dc1a78e724c8f9193b4 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/d2a67096-128f-4b8b-a31d-606f8eb8038f_isrForCrosstalkSources_1125933_16.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/d2a67096-128f-4b8b-a31d-606f8eb8038f_isrForCrosstalkSources_1125933_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:46 proc_lsst.multi:146 [INFO] found job 16896430 in provider escience_a40 proc_lsst.multi INFO: found job 16896430 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:47 proc_lsst.multi:146 [INFO] found job 16896431 in provider escience_a40 proc_lsst.multi INFO: found job 16896431 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4094401696e3c968e0b5c195e1ef50d9 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/5682fc07-95bb-4709-9f83-fec63990a7c4_isrForCrosstalkSources_1125903_48.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/5682fc07-95bb-4709-9f83-fec63990a7c4_isrForCrosstalkSources_1125903_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:47 proc_lsst.multi:146 [INFO] found job 16896432 in provider escience_a40 proc_lsst.multi INFO: found job 16896432 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:47 proc_lsst.multi:146 [INFO] found job 16896433 in provider escience_a40 proc_lsst.multi INFO: found job 16896433 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:48 proc_lsst.multi:146 [INFO] found job 16896434 in provider escience_a40 proc_lsst.multi INFO: found job 16896434 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:48 proc_lsst.multi:146 [INFO] found job 16896435 in provider escience_a40 proc_lsst.multi INFO: found job 16896435 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:49 proc_lsst.multi:146 [INFO] found job 16896437 in provider escience_a40 proc_lsst.multi INFO: found job 16896437 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:49 proc_lsst.multi:146 [INFO] found job 16896438 in provider escience_a40 proc_lsst.multi INFO: found job 16896438 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:50 proc_lsst.multi:146 [INFO] found job 16896439 in provider escience_a40 proc_lsst.multi INFO: found job 16896439 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:50 proc_lsst.multi:146 [INFO] found job 16896440 in provider escience_a40 proc_lsst.multi INFO: found job 16896440 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea454576ace87a71f29a3d9bdaabf8ae with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/f5b461e0-3b76-4ea2-9325-41cbf800cb25_isrForCrosstalkSources_1125915_32.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/f5b461e0-3b76-4ea2-9325-41cbf800cb25_isrForCrosstalkSources_1125915_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:51 proc_lsst.multi:146 [INFO] found job 16896441 in provider escience_a40 proc_lsst.multi INFO: found job 16896441 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:51 proc_lsst.multi:146 [INFO] found job 16896442 in provider escience_a40 proc_lsst.multi INFO: found job 16896442 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 053a31215cccbf27cb39ef59207bbebf with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9d093029-0178-466c-938b-6fb7008e4ced_isrForCrosstalkSources_1125921_20.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9d093029-0178-466c-938b-6fb7008e4ced_isrForCrosstalkSources_1125921_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:51 proc_lsst.multi:146 [INFO] found job 16896443 in provider escience_a40 proc_lsst.multi INFO: found job 16896443 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:52 proc_lsst.multi:146 [INFO] found job 16896444 in provider escience_a40 proc_lsst.multi INFO: found job 16896444 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77821b0ad70d095b1ddadd710e0f8015 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/7bf2f716-7445-4db4-8777-0142862d92b0_isrForCrosstalkSources_1125892_16.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/7bf2f716-7445-4db4-8777-0142862d92b0_isrForCrosstalkSources_1125892_16.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e92adce40120a569f114392243aa8ba with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/8506b516-2126-485e-a821-5b93d8f11d20_isrForCrosstalkSources_1125933_49.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/8506b516-2126-485e-a821-5b93d8f11d20_isrForCrosstalkSources_1125933_49.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fdd6878cc0bb6ba25892d8946318b8c with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/bdcf2eb2-ec37-4a83-88f4-8e4a98e56bf1_isrForCrosstalkSources_1125905_25.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/bdcf2eb2-ec37-4a83-88f4-8e4a98e56bf1_isrForCrosstalkSources_1125905_25.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9951a4a1678b402056b0839c8c050016 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/f0b33342-0063-455c-a915-3b381f62f51f_isrForCrosstalkSources_1125927_18.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/f0b33342-0063-455c-a915-3b381f62f51f_isrForCrosstalkSources_1125927_18.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc2c31c2fdf0ab0d8228fec073c1c7b6 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/25f86d2f-237b-4ba1-a704-777394b18320_isrForCrosstalkSources_1125894_14.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/25f86d2f-237b-4ba1-a704-777394b18320_isrForCrosstalkSources_1125894_14.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2649b71ea495ccf823ed7bf0791da8b4 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/fbdc29a2-3afb-4354-ad20-7028f874a58e_isrForCrosstalkSources_1125908_51.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/fbdc29a2-3afb-4354-ad20-7028f874a58e_isrForCrosstalkSources_1125908_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:07:52 proc_lsst.multi:146 [INFO] found job 16896445 in provider checkpoint proc_lsst.multi INFO: found job 16896445 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42f70afb086bfa8ee5c9cd9eb88865ee with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/c72de10f-cc60-4b34-99f5-3e82695f669a_isrForCrosstalkSources_1125900_56.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/c72de10f-cc60-4b34-99f5-3e82695f669a_isrForCrosstalkSources_1125900_56.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3216db4698eafc7165fc0968107b235f with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/4147bcdd-c20b-41c2-995f-b50aad399250_isrForCrosstalkSources_1125932_24.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/4147bcdd-c20b-41c2-995f-b50aad399250_isrForCrosstalkSources_1125932_24.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20b568341a1f3126cb104b6bb25d9417 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/61a54416-069f-4bab-81bc-1c732e5702f5_isrForCrosstalkSources_1125888_51.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/61a54416-069f-4bab-81bc-1c732e5702f5_isrForCrosstalkSources_1125888_51.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 311af6d7c5dcb6a08eaf8c66c3080e8e with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/accae294-1402-44ec-a3bc-3f439bb098d7_isrForCrosstalkSources_1125920_2.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/accae294-1402-44ec-a3bc-3f439bb098d7_isrForCrosstalkSources_1125920_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:07:53 proc_lsst.multi:146 [INFO] found job 16896446 in provider checkpoint proc_lsst.multi INFO: found job 16896446 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40dc44845f8ca21f69619ade179297ce with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/529eed37-d53f-4f39-9eba-45694d28a694_isrForCrosstalkSources_1125889_20.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/529eed37-d53f-4f39-9eba-45694d28a694_isrForCrosstalkSources_1125889_20.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d46b012fe13210f01eb12f30229509bc with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/abdb2c8d-8430-461d-a7ec-2fa8adf52f6e_isrForCrosstalkSources_1125943_12.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/abdb2c8d-8430-461d-a7ec-2fa8adf52f6e_isrForCrosstalkSources_1125943_12.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b348017eaefb105116961ea88c5f8533 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/5195510f-4f78-455f-8505-9199bbdd7578_isrForCrosstalkSources_1125907_44.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/5195510f-4f78-455f-8505-9199bbdd7578_isrForCrosstalkSources_1125907_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:07:53 proc_lsst.multi:146 [INFO] found job 16896447 in provider checkpoint proc_lsst.multi INFO: found job 16896447 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6279e848ef956de1d47f91009457707 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/bbb2d810-414c-4911-a9d9-9ca37a2437bb_isrForCrosstalkSources_1125890_47.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/bbb2d810-414c-4911-a9d9-9ca37a2437bb_isrForCrosstalkSources_1125890_47.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 438ca13030256bed0f3e0a689c375795 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/a9be5382-137e-46c1-b77a-6a1a20bf8a36_isrForCrosstalkSources_1125894_28.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/a9be5382-137e-46c1-b77a-6a1a20bf8a36_isrForCrosstalkSources_1125894_28.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff38e345b4f0fffbcf5662625ee9a4ec with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/778348fe-fdf7-4654-9985-72fe19bbadbc_isrForCrosstalkSources_1125923_14.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/778348fe-fdf7-4654-9985-72fe19bbadbc_isrForCrosstalkSources_1125923_14.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c65becfac8150f448cc9652c4e33e940 with result from task 63 parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:07:54 proc_lsst.multi:146 [INFO] found job 16896448 in provider checkpoint proc_lsst.multi INFO: found job 16896448 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/5df5e786-37af-455a-81c2-a898e65ef58e_isrForCrosstalkSources_1125891_13.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/5df5e786-37af-455a-81c2-a898e65ef58e_isrForCrosstalkSources_1125891_13.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d896420e5ee1b063d14b8912397f4a99 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/951642b6-34f7-4252-a481-bbac6180cdcf_isrForCrosstalkSources_1125890_38.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/951642b6-34f7-4252-a481-bbac6180cdcf_isrForCrosstalkSources_1125890_38.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5666b3a9d06e6a05f45ceced3cb4761b with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/0678c654-d0b0-42d7-b439-6505bc3089cb_isrForCrosstalkSources_1125913_39.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/0678c654-d0b0-42d7-b439-6505bc3089cb_isrForCrosstalkSources_1125913_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:07:54 proc_lsst.multi:146 [INFO] found job 16896449 in provider checkpoint proc_lsst.multi INFO: found job 16896449 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9222c887a684ddd82ad6f03e38f24fb4 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/582d64b6-e2db-43f0-8171-436c45e920e6_isrForCrosstalkSources_1125891_49.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/582d64b6-e2db-43f0-8171-436c45e920e6_isrForCrosstalkSources_1125891_49.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38fc20839fafc2dc3c578d28c9268ac1 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/1a07b8cd-1856-490c-9dcc-6a48de027825_isrForCrosstalkSources_1125903_20.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/1a07b8cd-1856-490c-9dcc-6a48de027825_isrForCrosstalkSources_1125903_20.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6adb746016bceea39082c0aaf1feb310 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/4871ec31-c4e0-4615-8853-307d9a9c0fba_isrForCrosstalkSources_1125921_5.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/4871ec31-c4e0-4615-8853-307d9a9c0fba_isrForCrosstalkSources_1125921_5.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3726f6879c6698d58aae4875e4226e0 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/8a95be13-6706-4a1d-96aa-23db49c7af77_isrForCrosstalkSources_1125908_7.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/8a95be13-6706-4a1d-96aa-23db49c7af77_isrForCrosstalkSources_1125908_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:07:55 proc_lsst.multi:146 [INFO] found job 16896450 in provider checkpoint proc_lsst.multi INFO: found job 16896450 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:07:55 proc_lsst.multi:146 [INFO] found job 16896451 in provider checkpoint proc_lsst.multi INFO: found job 16896451 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:07:55 proc_lsst.multi:146 [INFO] found job 16896452 in provider checkpoint proc_lsst.multi INFO: found job 16896452 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:07:55 proc_lsst.multi:146 [INFO] found job 16896453 in provider checkpoint proc_lsst.multi INFO: found job 16896453 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:07:56 proc_lsst.multi:146 [INFO] found job 16896454 in provider checkpoint proc_lsst.multi INFO: found job 16896454 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:07:56 proc_lsst.multi:146 [INFO] found job 16896455 in provider checkpoint proc_lsst.multi INFO: found job 16896455 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:07:56 proc_lsst.multi:146 [INFO] found job 16896456 in provider checkpoint proc_lsst.multi INFO: found job 16896456 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:07:57 proc_lsst.multi:146 [INFO] found job 16896457 in provider checkpoint proc_lsst.multi INFO: found job 16896457 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:07:57 proc_lsst.multi:146 [INFO] found job 16896458 in provider checkpoint proc_lsst.multi INFO: found job 16896458 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:07:58 proc_lsst.multi:146 [INFO] found job 16896459 in provider checkpoint proc_lsst.multi INFO: found job 16896459 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd462dfb7ef12347b0276442aa21fb4a with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f10d1891-5d74-470a-8fe1-d66dca0dc809_isrForCrosstalkSources_1125910_20.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f10d1891-5d74-470a-8fe1-d66dca0dc809_isrForCrosstalkSources_1125910_20.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7c0313a6fe3553f132a434dbe8218e2 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/5e1befb3-e89a-4eb1-9cac-696bf2fb6af3_isrForCrosstalkSources_1125897_31.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/5e1befb3-e89a-4eb1-9cac-696bf2fb6af3_isrForCrosstalkSources_1125897_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:00 proc_lsst.multi:146 [INFO] found job 16896460 in provider checkpoint proc_lsst.multi INFO: found job 16896460 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dc54788bd46d3220e79e787c398aae3 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/42d132d2-bb00-42b0-a24b-2bf9433d6b07_isrForCrosstalkSources_1125915_28.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/42d132d2-bb00-42b0-a24b-2bf9433d6b07_isrForCrosstalkSources_1125915_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:00 proc_lsst.multi:146 [INFO] found job 16896461 in provider checkpoint proc_lsst.multi INFO: found job 16896461 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:01 proc_lsst.multi:146 [INFO] found job 16896462 in provider checkpoint proc_lsst.multi INFO: found job 16896462 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a4ee75a8b5e2a30fe145766d497d30f with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/e76dd1d9-2ef7-4375-855d-0afc442024b7_isrForCrosstalkSources_1125936_19.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/e76dd1d9-2ef7-4375-855d-0afc442024b7_isrForCrosstalkSources_1125936_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:01 proc_lsst.multi:146 [INFO] found job 16896463 in provider checkpoint proc_lsst.multi INFO: found job 16896463 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 007c9ce3f8fc07c99efc86b75c077960 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/d3f290e4-89b0-4431-8141-b2031d291ef1_isrForCrosstalkSources_1125927_33.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/d3f290e4-89b0-4431-8141-b2031d291ef1_isrForCrosstalkSources_1125927_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:01 proc_lsst.multi:146 [INFO] found job 16896464 in provider checkpoint proc_lsst.multi INFO: found job 16896464 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12c7a1713dddb0d04b61746affade09c with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/fbab929d-ef93-41ef-abd6-35ac760bd0f4_isrForCrosstalkSources_1125916_57.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/fbab929d-ef93-41ef-abd6-35ac760bd0f4_isrForCrosstalkSources_1125916_57.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cce8905138973dd42a9a5ce00ee4d6a8 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/63b737cf-32b1-4025-ac1b-09f41b09bf68_isrForCrosstalkSources_1125928_50.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/63b737cf-32b1-4025-ac1b-09f41b09bf68_isrForCrosstalkSources_1125928_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:02 proc_lsst.multi:146 [INFO] found job 16896466 in provider checkpoint proc_lsst.multi INFO: found job 16896466 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:03 proc_lsst.multi:146 [INFO] found job 16896467 in provider checkpoint proc_lsst.multi INFO: found job 16896467 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e728234a692c6b9c8b26be21e078f5fd with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/29714b9d-4e9a-4b73-a00e-ffc2c5656506_isrForCrosstalkSources_1125893_13.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/29714b9d-4e9a-4b73-a00e-ffc2c5656506_isrForCrosstalkSources_1125893_13.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ae904777fc4dcb2e1471112902433e1 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/249acbbb-c9a3-4c8d-9ec4-b0fa96081c39_isrForCrosstalkSources_1125945_46.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/249acbbb-c9a3-4c8d-9ec4-b0fa96081c39_isrForCrosstalkSources_1125945_46.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a2c632b3731fd47f9b319171146f230 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/e1e2ad77-600a-4307-9656-32062a97c84a_isrForCrosstalkSources_1125917_48.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/e1e2ad77-600a-4307-9656-32062a97c84a_isrForCrosstalkSources_1125917_48.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc7e6208c9453bd9c3a764d5f7409dfd with result from task 84 parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:03 proc_lsst.multi:146 [INFO] found job 16896468 in provider checkpoint proc_lsst.multi INFO: found job 16896468 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/0a2fd52d-f7c6-4f7f-ab98-daa3376f84ae_isrForCrosstalkSources_1125892_29.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/0a2fd52d-f7c6-4f7f-ab98-daa3376f84ae_isrForCrosstalkSources_1125892_29.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d51ae8a8e28c521de735fdf04f1cfd2 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/8484f5dd-5d49-4e8e-b288-ccc53b214c10_isrForCrosstalkSources_1125894_34.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/8484f5dd-5d49-4e8e-b288-ccc53b214c10_isrForCrosstalkSources_1125894_34.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e0f743f003e40b91c9e175b58751979 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/736b7b58-60c4-48a1-bdfa-7691eac7db99_isrForCrosstalkSources_1125905_37.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/736b7b58-60c4-48a1-bdfa-7691eac7db99_isrForCrosstalkSources_1125905_37.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c17fb74512168226f8e8b62288e9e621 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/aeb828c8-d8f3-44ab-a120-e6dffdf4db1a_isrForCrosstalkSources_1125894_24.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/aeb828c8-d8f3-44ab-a120-e6dffdf4db1a_isrForCrosstalkSources_1125894_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:04 proc_lsst.multi:146 [INFO] found job 16896469 in provider checkpoint proc_lsst.multi INFO: found job 16896469 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51e5585abd0c7018b96071f0418b57fa with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/9894abba-05b4-4c7c-a07e-0c20b56bbcb3_isrForCrosstalkSources_1125932_47.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/9894abba-05b4-4c7c-a07e-0c20b56bbcb3_isrForCrosstalkSources_1125932_47.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 676424c298f95600c2f4a93f909e18cb with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/d3980308-2279-4232-9d95-7e2f3e322ef0_isrForCrosstalkSources_1125946_29.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/d3980308-2279-4232-9d95-7e2f3e322ef0_isrForCrosstalkSources_1125946_29.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4af414cf7c21b5881f0897c77219a0b0 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1a382e5e-caa5-43da-ab42-375b1932e1d4_isrForCrosstalkSources_1125900_54.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1a382e5e-caa5-43da-ab42-375b1932e1d4_isrForCrosstalkSources_1125900_54.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffe4d60877017046689f1b0bbef0598d with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/1880d524-f0a9-4bac-85d8-bbeeed80e33c_isrForCrosstalkSources_1125922_36.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/1880d524-f0a9-4bac-85d8-bbeeed80e33c_isrForCrosstalkSources_1125922_36.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0079216a0452e5f536cabd2fa94c8d61 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/2417ae47-05d9-40d6-bd34-679a1b06b9e9_isrForCrosstalkSources_1125891_48.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/2417ae47-05d9-40d6-bd34-679a1b06b9e9_isrForCrosstalkSources_1125891_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:04 proc_lsst.multi:146 [INFO] found job 16896470 in provider checkpoint proc_lsst.multi INFO: found job 16896470 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8061f6fdd5eaec561e8f759ba417e3b with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/46fa8c6f-ab64-4422-aca0-9383f4b53f77_isrForCrosstalkSources_1125903_62.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/46fa8c6f-ab64-4422-aca0-9383f4b53f77_isrForCrosstalkSources_1125903_62.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7fcacd79faa63724931817d292f1a71 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2af35943-fb81-484c-af44-4ed94e18fe54_isrForCrosstalkSources_1125895_35.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2af35943-fb81-484c-af44-4ed94e18fe54_isrForCrosstalkSources_1125895_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:05 proc_lsst.multi:146 [INFO] found job 16896471 in provider checkpoint proc_lsst.multi INFO: found job 16896471 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d85e8f00848890da4c9ae2fa07f3942c with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/252ef4f6-b1f2-4445-ad21-575fe3dfde8a_isrForCrosstalkSources_1125942_35.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/252ef4f6-b1f2-4445-ad21-575fe3dfde8a_isrForCrosstalkSources_1125942_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:05 proc_lsst.multi:146 [INFO] found job 16896472 in provider checkpoint proc_lsst.multi INFO: found job 16896472 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:05 proc_lsst.multi:146 [INFO] found job 16896473 in provider checkpoint proc_lsst.multi INFO: found job 16896473 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65267526e1e2ba04c2bfc2db5eb9e8a4 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/9a79402f-d2ed-4b41-b7fb-c1ceebaea66a_isrForCrosstalkSources_1125889_48.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/9a79402f-d2ed-4b41-b7fb-c1ceebaea66a_isrForCrosstalkSources_1125889_48.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85a39a90e1561089fd8e3a1a5aaf5e84 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d61392ee-1c21-4ffc-87f1-8b6bd20340ac_isrForCrosstalkSources_1125905_42.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d61392ee-1c21-4ffc-87f1-8b6bd20340ac_isrForCrosstalkSources_1125905_42.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a598cf3676767c1685293b7dddb59b7a with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3e4b0288-36ee-41b6-89f6-194b529a3302_isrForCrosstalkSources_1125912_48.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3e4b0288-36ee-41b6-89f6-194b529a3302_isrForCrosstalkSources_1125912_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:05 proc_lsst.multi:146 [INFO] found job 16896474 in provider checkpoint proc_lsst.multi INFO: found job 16896474 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d366b10fec08af1f5dade7a879738fa1 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/8c3c7b2d-d8ff-4e24-89c4-cd9577e1cf7b_isrForCrosstalkSources_1125890_61.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/8c3c7b2d-d8ff-4e24-89c4-cd9577e1cf7b_isrForCrosstalkSources_1125890_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:05 proc_lsst.multi:146 [INFO] found job 16896475 in provider checkpoint proc_lsst.multi INFO: found job 16896475 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:05 proc_lsst.multi:146 [INFO] found job 16896476 in provider checkpoint proc_lsst.multi INFO: found job 16896476 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61849744d915838a16122e3668d3d553 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/19e2f422-1aff-4747-b1a9-1615f84c065e_isrForCrosstalkSources_1125924_60.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/19e2f422-1aff-4747-b1a9-1615f84c065e_isrForCrosstalkSources_1125924_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:06 proc_lsst.multi:146 [INFO] found job 16896477 in provider checkpoint proc_lsst.multi INFO: found job 16896477 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:06 proc_lsst.multi:146 [INFO] found job 16896478 in provider checkpoint proc_lsst.multi INFO: found job 16896478 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:06 proc_lsst.multi:146 [INFO] found job 16896479 in provider checkpoint proc_lsst.multi INFO: found job 16896479 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:06 proc_lsst.multi:146 [INFO] found job 16896480 in provider checkpoint proc_lsst.multi INFO: found job 16896480 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:07 proc_lsst.multi:146 [INFO] found job 16896481 in provider checkpoint proc_lsst.multi INFO: found job 16896481 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:07 proc_lsst.multi:146 [INFO] found job 16896482 in provider checkpoint proc_lsst.multi INFO: found job 16896482 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:08 proc_lsst.multi:146 [INFO] found job 16896483 in provider checkpoint proc_lsst.multi INFO: found job 16896483 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:08 proc_lsst.multi:146 [INFO] found job 16896484 in provider checkpoint proc_lsst.multi INFO: found job 16896484 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:09 proc_lsst.multi:146 [INFO] found job 16896485 in provider checkpoint proc_lsst.multi INFO: found job 16896485 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:09 proc_lsst.multi:146 [INFO] found job 16896486 in provider checkpoint proc_lsst.multi INFO: found job 16896486 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:10 proc_lsst.multi:146 [INFO] found job 16896487 in provider checkpoint proc_lsst.multi INFO: found job 16896487 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3edf9d9c4ec794d67d5d55cdda12ff97 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/381c8b6e-4ea4-4660-80fa-494cc51aedbe_isrForCrosstalkSources_1125928_49.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/381c8b6e-4ea4-4660-80fa-494cc51aedbe_isrForCrosstalkSources_1125928_49.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49fbe96e087d6001e4189cf86f866125 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/636a962d-402a-4025-8ed3-9603c6607f85_isrForCrosstalkSources_1125904_30.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/636a962d-402a-4025-8ed3-9603c6607f85_isrForCrosstalkSources_1125904_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:10 proc_lsst.multi:146 [INFO] found job 16896488 in provider checkpoint proc_lsst.multi INFO: found job 16896488 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:10 proc_lsst.multi:146 [INFO] found job 16896489 in provider checkpoint proc_lsst.multi INFO: found job 16896489 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 562c175975034294ad63694e116c9a18 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/7c1a4e12-8ae7-4ba4-b09f-7ccc01dea38c_isrForCrosstalkSources_1125944_28.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/7c1a4e12-8ae7-4ba4-b09f-7ccc01dea38c_isrForCrosstalkSources_1125944_28.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a16774c79c412b9a1aba4f767f11e9a2 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/21cb01a3-d46c-4040-8126-d6cfe3693ec1_isrForCrosstalkSources_1125932_61.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/21cb01a3-d46c-4040-8126-d6cfe3693ec1_isrForCrosstalkSources_1125932_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:11 proc_lsst.multi:146 [INFO] found job 16896490 in provider checkpoint proc_lsst.multi INFO: found job 16896490 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:11 proc_lsst.multi:146 [INFO] found job 16896491 in provider checkpoint proc_lsst.multi INFO: found job 16896491 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e04185c7efb356f6680f376ff153e998 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/3e931932-d454-45b2-8f05-9fb450b99325_isrForCrosstalkSources_1125896_35.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/3e931932-d454-45b2-8f05-9fb450b99325_isrForCrosstalkSources_1125896_35.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4994d1432a49f51daf34f17f412b1b02 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f65cf5e4-9989-4013-ac78-cc14112b3fee_isrForCrosstalkSources_1125939_19.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f65cf5e4-9989-4013-ac78-cc14112b3fee_isrForCrosstalkSources_1125939_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:11 proc_lsst.multi:146 [INFO] found job 16896492 in provider checkpoint proc_lsst.multi INFO: found job 16896492 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:12 proc_lsst.multi:146 [INFO] found job 16896493 in provider checkpoint proc_lsst.multi INFO: found job 16896493 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:12 proc_lsst.multi:146 [INFO] found job 16896494 in provider checkpoint proc_lsst.multi INFO: found job 16896494 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:12 proc_lsst.multi:146 [INFO] found job 16896495 in provider checkpoint proc_lsst.multi INFO: found job 16896495 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:13 proc_lsst.multi:146 [INFO] found job 16896496 in provider checkpoint proc_lsst.multi INFO: found job 16896496 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a3435c03e0a873adb7e77c670145f2e with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/a2e5a877-1040-4edd-b5d8-bf6890bea9b3_isrForCrosstalkSources_1125926_57.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/a2e5a877-1040-4edd-b5d8-bf6890bea9b3_isrForCrosstalkSources_1125926_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:13 proc_lsst.multi:146 [INFO] found job 16896497 in provider checkpoint proc_lsst.multi INFO: found job 16896497 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:14 proc_lsst.multi:146 [INFO] found job 16896498 in provider checkpoint proc_lsst.multi INFO: found job 16896498 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:14 proc_lsst.multi:146 [INFO] found job 16896499 in provider checkpoint proc_lsst.multi INFO: found job 16896499 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c51bf804b0953dc87f6c096005d8aa60 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/68fe0192-4eda-4249-bbe9-f5ef133d6e86_isrForCrosstalkSources_1125903_25.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/68fe0192-4eda-4249-bbe9-f5ef133d6e86_isrForCrosstalkSources_1125903_25.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 803a6d91cc6e277fbd75e12f3001b338 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/74d27729-d5a2-449b-9fbc-8ffd00ebbbf9_isrForCrosstalkSources_1125916_26.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/74d27729-d5a2-449b-9fbc-8ffd00ebbbf9_isrForCrosstalkSources_1125916_26.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08f79a6a0caa97043a9451a8cad05c51 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a589e666-ef6a-4f13-ba7b-070b566b2825_isrForCrosstalkSources_1125938_25.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a589e666-ef6a-4f13-ba7b-070b566b2825_isrForCrosstalkSources_1125938_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:15 proc_lsst.multi:146 [INFO] found job 16896500 in provider checkpoint proc_lsst.multi INFO: found job 16896500 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:15 proc_lsst.multi:146 [INFO] found job 16896501 in provider checkpoint proc_lsst.multi INFO: found job 16896501 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e3d36a28da62575fc7b71e4fd2de9fb with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/efa687d1-59b9-4aa4-a62f-4212bbe46b89_isrForCrosstalkSources_1125945_4.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/efa687d1-59b9-4aa4-a62f-4212bbe46b89_isrForCrosstalkSources_1125945_4.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73f69cd58071d316c9cccbac22b6bc5f with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/897b9146-acf5-46b1-beab-f61822ddab22_isrForCrosstalkSources_1125888_8.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/897b9146-acf5-46b1-beab-f61822ddab22_isrForCrosstalkSources_1125888_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:15 proc_lsst.multi:146 [INFO] found job 16896502 in provider checkpoint proc_lsst.multi INFO: found job 16896502 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de239ce5170d73885830fb0cb36846e5 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/13852863-445f-421a-a151-dfbf3c8e7004_isrForCrosstalkSources_1125944_38.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/13852863-445f-421a-a151-dfbf3c8e7004_isrForCrosstalkSources_1125944_38.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3444a2f1d0d4c9ddf2e33ba9f9a5e6de with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/1d520af2-6392-4026-a1db-5188a396ffba_isrForCrosstalkSources_1125898_32.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/1d520af2-6392-4026-a1db-5188a396ffba_isrForCrosstalkSources_1125898_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:16 proc_lsst.multi:146 [INFO] found job 16896503 in provider checkpoint proc_lsst.multi INFO: found job 16896503 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2aa59227d72d0b643de2b1aeb068871 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/37420223-c466-42bd-85a9-7e4fcf048ae8_isrForCrosstalkSources_1125908_37.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/37420223-c466-42bd-85a9-7e4fcf048ae8_isrForCrosstalkSources_1125908_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:16 proc_lsst.multi:146 [INFO] found job 16896504 in provider checkpoint proc_lsst.multi INFO: found job 16896504 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cce2699bebec9639b36124b40b5c00ab with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/cd72e14d-3b21-42e5-8dd2-d4a7bf0c497d_isrForCrosstalkSources_1125917_3.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/cd72e14d-3b21-42e5-8dd2-d4a7bf0c497d_isrForCrosstalkSources_1125917_3.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba30123a0e21ce56c37ce34801fd3719 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/67a3ed79-c8c7-4a34-a77b-c0f1e1086f2a_isrForCrosstalkSources_1125911_7.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/67a3ed79-c8c7-4a34-a77b-c0f1e1086f2a_isrForCrosstalkSources_1125911_7.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a55fb83a570a2c266e5da52ea08ffb6 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/99a057c7-ea39-464d-acc3-7fd8ace0ee62_isrForCrosstalkSources_1125924_6.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/99a057c7-ea39-464d-acc3-7fd8ace0ee62_isrForCrosstalkSources_1125924_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:16 proc_lsst.multi:146 [INFO] found job 16896505 in provider checkpoint proc_lsst.multi INFO: found job 16896505 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c6e139424fd13ea5608a885e6141fb0 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/34a6cc6e-aaa4-4922-8227-637542c557b3_isrForCrosstalkSources_1125925_36.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/34a6cc6e-aaa4-4922-8227-637542c557b3_isrForCrosstalkSources_1125925_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:16 proc_lsst.multi:146 [INFO] found job 16896506 in provider checkpoint proc_lsst.multi INFO: found job 16896506 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa78a33f81601d0ad3c2400a87c40b9e with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/1e75d3be-021d-4e40-b512-77105f6b6edc_isrForCrosstalkSources_1125929_7.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/1e75d3be-021d-4e40-b512-77105f6b6edc_isrForCrosstalkSources_1125929_7.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72229eb25c5a65a16b40aefae96ad172 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/10fc6be7-9d63-4d5a-b5c2-d5a3170e761b_isrForCrosstalkSources_1125896_44.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/10fc6be7-9d63-4d5a-b5c2-d5a3170e761b_isrForCrosstalkSources_1125896_44.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97f022d97f4ddbec711b8008b2573e1e with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/35ad8c02-1891-4f99-a4b8-c5d7040723fc_isrForCrosstalkSources_1125944_61.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/35ad8c02-1891-4f99-a4b8-c5d7040723fc_isrForCrosstalkSources_1125944_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:17 proc_lsst.multi:146 [INFO] found job 16896507 in provider checkpoint proc_lsst.multi INFO: found job 16896507 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24fbbc8eb50f22ec9f55e325c6a69179 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/732a12a3-8689-41b8-9394-c7793c90f2f6_isrForCrosstalkSources_1125891_29.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/732a12a3-8689-41b8-9394-c7793c90f2f6_isrForCrosstalkSources_1125891_29.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d9f6ac451ac86010f5d111e6ebb77d1 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/d1f52508-f22e-49fa-8ac3-217af1ea50db_isrForCrosstalkSources_1125907_42.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/d1f52508-f22e-49fa-8ac3-217af1ea50db_isrForCrosstalkSources_1125907_42.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c23d176a4dfaa956fca8936f13171d39 with result from task 107 parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:17 proc_lsst.multi:146 [INFO] found job 16896508 in provider checkpoint proc_lsst.multi INFO: found job 16896508 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/870ed48f-1f68-4f87-88c2-6d3ad77e2476_isrForCrosstalkSources_1125894_53.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/870ed48f-1f68-4f87-88c2-6d3ad77e2476_isrForCrosstalkSources_1125894_53.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be31080a12f1fa8129805ecc6ed5ee9b with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/34d27b8d-31a6-4370-b138-8119fa0d6551_isrForCrosstalkSources_1125902_59.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/34d27b8d-31a6-4370-b138-8119fa0d6551_isrForCrosstalkSources_1125902_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:18 proc_lsst.multi:146 [INFO] found job 16896509 in provider checkpoint proc_lsst.multi INFO: found job 16896509 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:18 proc_lsst.multi:146 [INFO] found job 16896510 in provider checkpoint proc_lsst.multi INFO: found job 16896510 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2197304b6c34b1f7265cca6567cf8c2 with result from task 121 parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING 2024-03-07 15:08:19 proc_lsst.multi:146 [INFO] found job 16896511 in provider checkpoint proc_lsst.multi INFO: found job 16896511 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511' parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/982e2482-dcab-4d3b-8f72-38ce4ee416ef_isrForCrosstalkSources_1125919_29.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/982e2482-dcab-4d3b-8f72-38ce4ee416ef_isrForCrosstalkSources_1125919_29.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10b9d2f2e029b0c46aca79cf9e3d287e with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/2fba4068-59f9-42ea-bf7f-f583d6490996_isrForCrosstalkSources_1125893_16.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/2fba4068-59f9-42ea-bf7f-f583d6490996_isrForCrosstalkSources_1125893_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896445 PD 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 122, active_tasks = 3601 parsl.dataflow.strategy DEBUG: Executor multi has 3601 active tasks, 34/88 running/pending blocks, and 35 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 122 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 190 more blocks parsl.executors.status_handling INFO: Scaling out by 190 blocks parsl.executors.status_handling INFO: Allocated block ID 122 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-122 2024-03-07 15:08:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-122 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-122 2024-03-07 15:08:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 66 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 66 2024-03-07 15:08:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee2df4278b25db4f7b6bee54e1de9da4 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/c0c14190-d1f9-4dbb-9473-dba12b43f2f2_isrForCrosstalkSources_1125933_23.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/c0c14190-d1f9-4dbb-9473-dba12b43f2f2_isrForCrosstalkSources_1125933_23.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 349ead40ddb1ad0cd4770d91d8e9c104 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/b3c27f80-46ab-4c1b-92d2-d15d4cc23ff5_isrForCrosstalkSources_1125903_36.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/b3c27f80-46ab-4c1b-92d2-d15d4cc23ff5_isrForCrosstalkSources_1125903_36.stderr 2024-03-07 15:08:20 proc_lsst.multi:170 [INFO] [multi] job_id 16896519 proc_lsst.multi INFO: [multi] job_id 16896519 2024-03-07 15:08:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 67 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 67 2024-03-07 15:08:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896519 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896519 parsl.executors.status_handling DEBUG: Launched block 122 on executor multi with job ID 16896519 parsl.executors.status_handling INFO: Allocated block ID 123 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-123 2024-03-07 15:08:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-123 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-123 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 67 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 67 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:20 proc_lsst.multi:170 [INFO] [multi] job_id 16896520 proc_lsst.multi INFO: [multi] job_id 16896520 2024-03-07 15:08:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 68 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 68 2024-03-07 15:08:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896520 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896520 parsl.executors.status_handling DEBUG: Launched block 123 on executor multi with job ID 16896520 parsl.executors.status_handling INFO: Allocated block ID 124 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-124 2024-03-07 15:08:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-124 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-124 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 68 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 68 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:20 proc_lsst.multi:170 [INFO] [multi] job_id 16896521 proc_lsst.multi INFO: [multi] job_id 16896521 2024-03-07 15:08:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 69 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 69 2024-03-07 15:08:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896521 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896521 parsl.executors.status_handling DEBUG: Launched block 124 on executor multi with job ID 16896521 parsl.executors.status_handling INFO: Allocated block ID 125 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-125 2024-03-07 15:08:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-125 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-125 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 69 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 69 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:20 proc_lsst.multi:170 [INFO] [multi] job_id 16896522 proc_lsst.multi INFO: [multi] job_id 16896522 2024-03-07 15:08:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 70 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 70 2024-03-07 15:08:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896522 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896522 parsl.executors.status_handling DEBUG: Launched block 125 on executor multi with job ID 16896522 parsl.executors.status_handling INFO: Allocated block ID 126 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-126 2024-03-07 15:08:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-126 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-126 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 70 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 70 2024-03-07 15:08:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8954f1ba26dde7c9a1975481826d331 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/a42b73ef-2779-4d2a-9043-acf31f91777b_isrForCrosstalkSources_1125902_17.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/a42b73ef-2779-4d2a-9043-acf31f91777b_isrForCrosstalkSources_1125902_17.stderr 2024-03-07 15:08:21 proc_lsst.multi:170 [INFO] [multi] job_id 16896523 proc_lsst.multi INFO: [multi] job_id 16896523 2024-03-07 15:08:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 71 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 71 2024-03-07 15:08:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896523 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896523 parsl.executors.status_handling DEBUG: Launched block 126 on executor multi with job ID 16896523 parsl.executors.status_handling INFO: Allocated block ID 127 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-127 2024-03-07 15:08:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-127 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-127 2024-03-07 15:08:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 71 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 71 2024-03-07 15:08:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:21 proc_lsst.multi:170 [INFO] [multi] job_id 16896524 proc_lsst.multi INFO: [multi] job_id 16896524 2024-03-07 15:08:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 72 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 72 2024-03-07 15:08:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896524 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896524 parsl.executors.status_handling DEBUG: Launched block 127 on executor multi with job ID 16896524 parsl.executors.status_handling INFO: Allocated block ID 128 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-128 2024-03-07 15:08:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-128 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-128 2024-03-07 15:08:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 72 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 72 2024-03-07 15:08:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07a298345eaed794e1a3014a42ed9e9a with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/7cca0f2e-7898-46ac-94e5-0b81b8a6eb7d_isrForCrosstalkSources_1125898_58.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/7cca0f2e-7898-46ac-94e5-0b81b8a6eb7d_isrForCrosstalkSources_1125898_58.stderr 2024-03-07 15:08:21 proc_lsst.multi:170 [INFO] [multi] job_id 16896525 proc_lsst.multi INFO: [multi] job_id 16896525 2024-03-07 15:08:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 73 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 73 2024-03-07 15:08:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896525 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896525 parsl.executors.status_handling DEBUG: Launched block 128 on executor multi with job ID 16896525 parsl.executors.status_handling INFO: Allocated block ID 129 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-129 2024-03-07 15:08:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-129 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-129 2024-03-07 15:08:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 73 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 73 2024-03-07 15:08:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:22 proc_lsst.multi:170 [INFO] [multi] job_id 16896526 proc_lsst.multi INFO: [multi] job_id 16896526 2024-03-07 15:08:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 74 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 74 2024-03-07 15:08:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896526 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896526 parsl.executors.status_handling DEBUG: Launched block 129 on executor multi with job ID 16896526 parsl.executors.status_handling INFO: Allocated block ID 130 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-130 2024-03-07 15:08:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-130 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-130 2024-03-07 15:08:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 74 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 74 2024-03-07 15:08:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:22 proc_lsst.multi:170 [INFO] [multi] job_id 16896527 proc_lsst.multi INFO: [multi] job_id 16896527 2024-03-07 15:08:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 75 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 75 2024-03-07 15:08:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896527 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896527 parsl.executors.status_handling DEBUG: Launched block 130 on executor multi with job ID 16896527 parsl.executors.status_handling INFO: Allocated block ID 131 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-131 2024-03-07 15:08:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-131 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-131 2024-03-07 15:08:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 75 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 75 2024-03-07 15:08:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e03a5156bb0a87eb360b071e3bec1f4 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/b9aa431f-8f22-4de1-b0ba-1046641bb122_isrForCrosstalkSources_1125943_32.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/b9aa431f-8f22-4de1-b0ba-1046641bb122_isrForCrosstalkSources_1125943_32.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61479cfeafe35e266f86be5af3b068f1 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/9fade219-c30d-4c4a-ab2d-24e5da8f7dd7_isrForCrosstalkSources_1125902_34.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/9fade219-c30d-4c4a-ab2d-24e5da8f7dd7_isrForCrosstalkSources_1125902_34.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 908b0a7feafb27aad2736db134c36fbb with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/b4962d4b-7344-4b84-82a3-645692160406_isrForCrosstalkSources_1125931_36.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/b4962d4b-7344-4b84-82a3-645692160406_isrForCrosstalkSources_1125931_36.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 715a10e97632c8d1eb4debd30778705c with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/e518c075-ad88-45c4-9cb5-dde35bd25204_isrForCrosstalkSources_1125915_6.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/e518c075-ad88-45c4-9cb5-dde35bd25204_isrForCrosstalkSources_1125915_6.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91a7cd5e3980ee86fdaa2c87f45a8703 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8e71bdda-4b2a-410a-abc2-f9f75bff77e5_isrForCrosstalkSources_1125944_19.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8e71bdda-4b2a-410a-abc2-f9f75bff77e5_isrForCrosstalkSources_1125944_19.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5dd8dfc36a1f9ebd08e9554eb6daac7 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/0994fd91-c49c-4440-a3ba-53f052d1fe55_isrForCrosstalkSources_1125891_59.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/0994fd91-c49c-4440-a3ba-53f052d1fe55_isrForCrosstalkSources_1125891_59.stderr 2024-03-07 15:08:22 proc_lsst.multi:170 [INFO] [multi] job_id 16896528 proc_lsst.multi INFO: [multi] job_id 16896528 2024-03-07 15:08:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 76 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 76 2024-03-07 15:08:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896528 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896528 parsl.executors.status_handling DEBUG: Launched block 131 on executor multi with job ID 16896528 parsl.executors.status_handling INFO: Allocated block ID 132 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-132 2024-03-07 15:08:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-132 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-132 2024-03-07 15:08:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 76 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 76 2024-03-07 15:08:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:23 proc_lsst.multi:170 [INFO] [multi] job_id 16896529 proc_lsst.multi INFO: [multi] job_id 16896529 2024-03-07 15:08:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 77 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 77 2024-03-07 15:08:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896529 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896529 parsl.executors.status_handling DEBUG: Launched block 132 on executor multi with job ID 16896529 parsl.executors.status_handling INFO: Allocated block ID 133 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-133 2024-03-07 15:08:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-133 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-133 2024-03-07 15:08:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 77 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 77 2024-03-07 15:08:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:23 proc_lsst.multi:170 [INFO] [multi] job_id 16896530 proc_lsst.multi INFO: [multi] job_id 16896530 2024-03-07 15:08:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 78 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 78 2024-03-07 15:08:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896530 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896530 parsl.executors.status_handling DEBUG: Launched block 133 on executor multi with job ID 16896530 parsl.executors.status_handling INFO: Allocated block ID 134 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-134 2024-03-07 15:08:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-134 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-134 2024-03-07 15:08:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 78 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 78 2024-03-07 15:08:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10aad420b8934d106c3f0a727f88629f with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/23a959a7-3b1e-4cdc-9eb9-cc25f98f196e_isrForCrosstalkSources_1125911_61.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/23a959a7-3b1e-4cdc-9eb9-cc25f98f196e_isrForCrosstalkSources_1125911_61.stderr 2024-03-07 15:08:23 proc_lsst.multi:170 [INFO] [multi] job_id 16896531 proc_lsst.multi INFO: [multi] job_id 16896531 2024-03-07 15:08:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 79 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 79 2024-03-07 15:08:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896531 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896531 parsl.executors.status_handling DEBUG: Launched block 134 on executor multi with job ID 16896531 parsl.executors.status_handling INFO: Allocated block ID 135 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-135 2024-03-07 15:08:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-135 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-135 2024-03-07 15:08:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 79 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 79 2024-03-07 15:08:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b70d92d14fa42efdc81be920fea36e59 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/27d815e8-e864-4b3d-a915-8f927d9d8709_isrForCrosstalkSources_1125929_19.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/27d815e8-e864-4b3d-a915-8f927d9d8709_isrForCrosstalkSources_1125929_19.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05aed38672b54a3b3395ff47e1c3fa0f with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/c4c6e8de-0e08-4ec6-bb0a-536c09ff69fe_isrForCrosstalkSources_1125926_53.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/c4c6e8de-0e08-4ec6-bb0a-536c09ff69fe_isrForCrosstalkSources_1125926_53.stderr 2024-03-07 15:08:24 proc_lsst.multi:170 [INFO] [multi] job_id 16896533 proc_lsst.multi INFO: [multi] job_id 16896533 2024-03-07 15:08:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 80 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 80 2024-03-07 15:08:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896533 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896533 parsl.executors.status_handling DEBUG: Launched block 135 on executor multi with job ID 16896533 parsl.executors.status_handling INFO: Allocated block ID 136 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-136 2024-03-07 15:08:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-136 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-136 2024-03-07 15:08:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 80 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 80 2024-03-07 15:08:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:24 proc_lsst.multi:170 [INFO] [multi] job_id 16896534 proc_lsst.multi INFO: [multi] job_id 16896534 2024-03-07 15:08:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 81 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 81 2024-03-07 15:08:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896534 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896534 parsl.executors.status_handling DEBUG: Launched block 136 on executor multi with job ID 16896534 parsl.executors.status_handling INFO: Allocated block ID 137 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-137 2024-03-07 15:08:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-137 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-137 2024-03-07 15:08:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 81 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 81 2024-03-07 15:08:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:24 proc_lsst.multi:170 [INFO] [multi] job_id 16896535 proc_lsst.multi INFO: [multi] job_id 16896535 2024-03-07 15:08:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 82 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 82 2024-03-07 15:08:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896535 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896535 parsl.executors.status_handling DEBUG: Launched block 137 on executor multi with job ID 16896535 parsl.executors.status_handling INFO: Allocated block ID 138 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-138 2024-03-07 15:08:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-138 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-138 2024-03-07 15:08:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 82 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 82 2024-03-07 15:08:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:25 proc_lsst.multi:170 [INFO] [multi] job_id 16896536 proc_lsst.multi INFO: [multi] job_id 16896536 2024-03-07 15:08:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 83 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 83 2024-03-07 15:08:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896536 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896536 parsl.executors.status_handling DEBUG: Launched block 138 on executor multi with job ID 16896536 parsl.executors.status_handling INFO: Allocated block ID 139 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-139 2024-03-07 15:08:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-139 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-139 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 83 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 83 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:25 proc_lsst.multi:170 [INFO] [multi] job_id 16896537 proc_lsst.multi INFO: [multi] job_id 16896537 2024-03-07 15:08:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 84 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 84 2024-03-07 15:08:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896537 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896537 parsl.executors.status_handling DEBUG: Launched block 139 on executor multi with job ID 16896537 parsl.executors.status_handling INFO: Allocated block ID 140 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-140 2024-03-07 15:08:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-140 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-140 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 84 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 84 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:25 proc_lsst.multi:170 [INFO] [multi] job_id 16896538 proc_lsst.multi INFO: [multi] job_id 16896538 2024-03-07 15:08:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 85 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 85 2024-03-07 15:08:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896538 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896538 parsl.executors.status_handling DEBUG: Launched block 140 on executor multi with job ID 16896538 parsl.executors.status_handling INFO: Allocated block ID 141 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-141 2024-03-07 15:08:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-141 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-141 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 85 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 85 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:25 proc_lsst.multi:170 [INFO] [multi] job_id 16896539 proc_lsst.multi INFO: [multi] job_id 16896539 2024-03-07 15:08:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 86 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 86 2024-03-07 15:08:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896539 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896539 parsl.executors.status_handling DEBUG: Launched block 141 on executor multi with job ID 16896539 parsl.executors.status_handling INFO: Allocated block ID 142 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-142 2024-03-07 15:08:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-142 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-142 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 86 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 86 2024-03-07 15:08:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:26 proc_lsst.multi:170 [INFO] [multi] job_id 16896540 proc_lsst.multi INFO: [multi] job_id 16896540 2024-03-07 15:08:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 87 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 87 2024-03-07 15:08:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896540 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896540 parsl.executors.status_handling DEBUG: Launched block 142 on executor multi with job ID 16896540 parsl.executors.status_handling INFO: Allocated block ID 143 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-143 2024-03-07 15:08:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-143 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-143 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 87 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 87 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a003a0d0941997cf501d29b4bedc795d with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ce6a4b7a-b1d0-4f2c-a75e-8c59302d2786_isrForCrosstalkSources_1125894_27.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ce6a4b7a-b1d0-4f2c-a75e-8c59302d2786_isrForCrosstalkSources_1125894_27.stderr 2024-03-07 15:08:26 proc_lsst.multi:170 [INFO] [multi] job_id 16896541 proc_lsst.multi INFO: [multi] job_id 16896541 2024-03-07 15:08:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 88 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 88 2024-03-07 15:08:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896541 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896541 parsl.executors.status_handling DEBUG: Launched block 143 on executor multi with job ID 16896541 parsl.executors.status_handling INFO: Allocated block ID 144 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-144 2024-03-07 15:08:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-144 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-144 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 88 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 88 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:26 proc_lsst.multi:170 [INFO] [multi] job_id 16896542 proc_lsst.multi INFO: [multi] job_id 16896542 2024-03-07 15:08:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 89 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 89 2024-03-07 15:08:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896542 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896542 parsl.executors.status_handling DEBUG: Launched block 144 on executor multi with job ID 16896542 parsl.executors.status_handling INFO: Allocated block ID 145 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-145 2024-03-07 15:08:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-145 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-145 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 89 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 89 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24531f05ccc6918c9787cc0c3306cd38 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/c48044fa-0ed4-4888-848d-6306c4595c8d_isrForCrosstalkSources_1125940_52.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/c48044fa-0ed4-4888-848d-6306c4595c8d_isrForCrosstalkSources_1125940_52.stderr 2024-03-07 15:08:26 proc_lsst.multi:170 [INFO] [multi] job_id 16896543 proc_lsst.multi INFO: [multi] job_id 16896543 2024-03-07 15:08:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 90 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 90 2024-03-07 15:08:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896543 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896543 parsl.executors.status_handling DEBUG: Launched block 145 on executor multi with job ID 16896543 parsl.executors.status_handling INFO: Allocated block ID 146 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-146 2024-03-07 15:08:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-146 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-146 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 90 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 90 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:26 proc_lsst.multi:170 [INFO] [multi] job_id 16896544 proc_lsst.multi INFO: [multi] job_id 16896544 2024-03-07 15:08:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 91 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 91 2024-03-07 15:08:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896544 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896544 parsl.executors.status_handling DEBUG: Launched block 146 on executor multi with job ID 16896544 parsl.executors.status_handling INFO: Allocated block ID 147 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-147 2024-03-07 15:08:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-147 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-147 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 91 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 91 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:26 proc_lsst.multi:170 [INFO] [multi] job_id 16896545 proc_lsst.multi INFO: [multi] job_id 16896545 2024-03-07 15:08:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 92 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 92 2024-03-07 15:08:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896545 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896545 parsl.executors.status_handling DEBUG: Launched block 147 on executor multi with job ID 16896545 parsl.executors.status_handling INFO: Allocated block ID 148 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-148 2024-03-07 15:08:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-148 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-148 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 92 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 92 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67fcb8b289c3d802a833a6f69adab0e9 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/f70fbca4-fe7b-4d1a-a7dc-fc2fcbc7f7fd_isrForCrosstalkSources_1125932_20.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/f70fbca4-fe7b-4d1a-a7dc-fc2fcbc7f7fd_isrForCrosstalkSources_1125932_20.stderr 2024-03-07 15:08:26 proc_lsst.multi:170 [INFO] [multi] job_id 16896546 proc_lsst.multi INFO: [multi] job_id 16896546 2024-03-07 15:08:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 93 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 93 2024-03-07 15:08:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896546 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896546 parsl.executors.status_handling DEBUG: Launched block 148 on executor multi with job ID 16896546 parsl.executors.status_handling INFO: Allocated block ID 149 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-149 2024-03-07 15:08:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-149 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-149 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 93 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 93 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:26 proc_lsst.multi:170 [INFO] [multi] job_id 16896547 proc_lsst.multi INFO: [multi] job_id 16896547 2024-03-07 15:08:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 94 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 94 2024-03-07 15:08:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896547 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896547 parsl.executors.status_handling DEBUG: Launched block 149 on executor multi with job ID 16896547 parsl.executors.status_handling INFO: Allocated block ID 150 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-150 2024-03-07 15:08:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-150 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-150 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 94 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 94 2024-03-07 15:08:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a20e22245747195190014469140dc5c with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/12eaf205-8d78-430a-b089-b9683afe48c9_isrForCrosstalkSources_1125929_23.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/12eaf205-8d78-430a-b089-b9683afe48c9_isrForCrosstalkSources_1125929_23.stderr 2024-03-07 15:08:27 proc_lsst.multi:170 [INFO] [multi] job_id 16896548 proc_lsst.multi INFO: [multi] job_id 16896548 2024-03-07 15:08:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 95 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 95 2024-03-07 15:08:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896548 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896548 parsl.executors.status_handling DEBUG: Launched block 150 on executor multi with job ID 16896548 parsl.executors.status_handling INFO: Allocated block ID 151 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-151 2024-03-07 15:08:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-151 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-151 2024-03-07 15:08:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 95 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 95 2024-03-07 15:08:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4fa86c45a78a7d208fda23242dac255 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f67891c5-343d-4156-a7e2-97b10272d62f_isrForCrosstalkSources_1125930_18.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f67891c5-343d-4156-a7e2-97b10272d62f_isrForCrosstalkSources_1125930_18.stderr 2024-03-07 15:08:27 proc_lsst.multi:170 [INFO] [multi] job_id 16896549 proc_lsst.multi INFO: [multi] job_id 16896549 2024-03-07 15:08:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 96 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 96 2024-03-07 15:08:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896549 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896549 parsl.executors.status_handling DEBUG: Launched block 151 on executor multi with job ID 16896549 parsl.executors.status_handling INFO: Allocated block ID 152 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-152 2024-03-07 15:08:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-152 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-152 2024-03-07 15:08:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 96 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 96 2024-03-07 15:08:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecbcd1bc1c1905a366d8ad3f9de800a2 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9fe71f8f-5b6d-41e6-bdd0-8a0bf91671bd_isrForCrosstalkSources_1125921_3.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9fe71f8f-5b6d-41e6-bdd0-8a0bf91671bd_isrForCrosstalkSources_1125921_3.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cd96ce6fcbc26010181d96b16581fbd with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/45ab2447-cfef-488a-946a-5a0920843ca4_isrForCrosstalkSources_1125916_31.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/45ab2447-cfef-488a-946a-5a0920843ca4_isrForCrosstalkSources_1125916_31.stderr 2024-03-07 15:08:27 proc_lsst.multi:170 [INFO] [multi] job_id 16896550 proc_lsst.multi INFO: [multi] job_id 16896550 2024-03-07 15:08:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 97 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 97 2024-03-07 15:08:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896550 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896550 parsl.executors.status_handling DEBUG: Launched block 152 on executor multi with job ID 16896550 parsl.executors.status_handling INFO: Allocated block ID 153 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-153 2024-03-07 15:08:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-153 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-153 2024-03-07 15:08:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 97 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 97 2024-03-07 15:08:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:28 proc_lsst.multi:170 [INFO] [multi] job_id 16896551 proc_lsst.multi INFO: [multi] job_id 16896551 2024-03-07 15:08:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 98 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 98 2024-03-07 15:08:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896551 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896551 parsl.executors.status_handling DEBUG: Launched block 153 on executor multi with job ID 16896551 parsl.executors.status_handling INFO: Allocated block ID 154 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-154 2024-03-07 15:08:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-154 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-154 2024-03-07 15:08:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 98 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 98 2024-03-07 15:08:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6b65e1df6f9ab66b3603435c587bbe8 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/d23de85d-acde-49c3-a701-8ffb2635b2e9_isrForCrosstalkSources_1125904_4.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/d23de85d-acde-49c3-a701-8ffb2635b2e9_isrForCrosstalkSources_1125904_4.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44dea3ab1b3fca0c292e0ef33bcaef2b with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/6870bee0-c350-4c9c-8233-816a24383c3a_isrForCrosstalkSources_1125897_21.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/6870bee0-c350-4c9c-8233-816a24383c3a_isrForCrosstalkSources_1125897_21.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e673574017ba00184ed6a3de916182c with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/3ec75d67-7c51-4b09-9610-850f41f368db_isrForCrosstalkSources_1125913_53.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/3ec75d67-7c51-4b09-9610-850f41f368db_isrForCrosstalkSources_1125913_53.stderr 2024-03-07 15:08:28 proc_lsst.multi:170 [INFO] [multi] job_id 16896552 proc_lsst.multi INFO: [multi] job_id 16896552 2024-03-07 15:08:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 99 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 99 2024-03-07 15:08:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896552 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896552 parsl.executors.status_handling DEBUG: Launched block 154 on executor multi with job ID 16896552 parsl.executors.status_handling INFO: Allocated block ID 155 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-155 2024-03-07 15:08:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-155 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-155 2024-03-07 15:08:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 99 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 99 2024-03-07 15:08:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cddbfff789d4839d5131bac0869e8634 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/14177429-dd95-48cd-9e78-af1ba5d9da8d_isrForCrosstalkSources_1125888_53.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/14177429-dd95-48cd-9e78-af1ba5d9da8d_isrForCrosstalkSources_1125888_53.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c2fbc2f6881e76a9191b4944c12e950 with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/352ef2fe-fca3-4285-aa2b-29ef21210ce6_isrForCrosstalkSources_1125931_59.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/352ef2fe-fca3-4285-aa2b-29ef21210ce6_isrForCrosstalkSources_1125931_59.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7db7eebf58c6781e8aeef58aab3462d with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/b5a32698-416f-4680-8ee9-d0bf0679ccea_isrForCrosstalkSources_1125913_15.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/b5a32698-416f-4680-8ee9-d0bf0679ccea_isrForCrosstalkSources_1125913_15.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcb4537f37e882ad7e5f2d12673b0531 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/1cd98e05-1f80-4412-a39b-fef696d6009b_isrForCrosstalkSources_1125916_8.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/1cd98e05-1f80-4412-a39b-fef696d6009b_isrForCrosstalkSources_1125916_8.stderr 2024-03-07 15:08:29 proc_lsst.multi:170 [INFO] [multi] job_id 16896553 proc_lsst.multi INFO: [multi] job_id 16896553 2024-03-07 15:08:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 100 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 100 2024-03-07 15:08:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896553 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896553 parsl.executors.status_handling DEBUG: Launched block 155 on executor multi with job ID 16896553 parsl.executors.status_handling INFO: Allocated block ID 156 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-156 2024-03-07 15:08:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-156 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-156 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 100 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 100 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c41c04c9342949fc31a620e92b350097 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/b3c82e66-8cf8-4f89-a625-f6a47b1304b5_isrForCrosstalkSources_1125910_32.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/b3c82e66-8cf8-4f89-a625-f6a47b1304b5_isrForCrosstalkSources_1125910_32.stderr 2024-03-07 15:08:29 proc_lsst.multi:170 [INFO] [multi] job_id 16896554 proc_lsst.multi INFO: [multi] job_id 16896554 2024-03-07 15:08:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 101 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 101 2024-03-07 15:08:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896554 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896554 parsl.executors.status_handling DEBUG: Launched block 156 on executor multi with job ID 16896554 parsl.executors.status_handling INFO: Allocated block ID 157 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-157 2024-03-07 15:08:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-157 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-157 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 101 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 101 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 723990e198c7b64e0f84103999f98034 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/02fc6273-18c7-4a2b-bd99-81df2df7cfca_isrForCrosstalkSources_1125891_9.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/02fc6273-18c7-4a2b-bd99-81df2df7cfca_isrForCrosstalkSources_1125891_9.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a36b6681fae366f7b1054cd1e0853d7a with result from task 136 2024-03-07 15:08:29 proc_lsst.multi:170 [INFO] [multi] job_id 16896555 proc_lsst.multi INFO: [multi] job_id 16896555 2024-03-07 15:08:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 102 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 102 2024-03-07 15:08:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896555 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896555 parsl.executors.status_handling DEBUG: Launched block 157 on executor multi with job ID 16896555 parsl.executors.status_handling INFO: Allocated block ID 158 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-158 2024-03-07 15:08:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-158 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-158 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 102 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 102 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/c70831fd-bc2b-4646-b39f-b0dfcefb2eaf_isrForCrosstalkSources_1125940_10.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/c70831fd-bc2b-4646-b39f-b0dfcefb2eaf_isrForCrosstalkSources_1125940_10.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00091618b6ff5df83a4def6d8d7026b7 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/2169826b-3702-4aca-8ab8-faee5e21aac4_isrForCrosstalkSources_1125914_35.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/2169826b-3702-4aca-8ab8-faee5e21aac4_isrForCrosstalkSources_1125914_35.stderr 2024-03-07 15:08:29 proc_lsst.multi:170 [INFO] [multi] job_id 16896556 proc_lsst.multi INFO: [multi] job_id 16896556 2024-03-07 15:08:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 103 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 103 2024-03-07 15:08:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896556 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896556 parsl.executors.status_handling DEBUG: Launched block 158 on executor multi with job ID 16896556 parsl.executors.status_handling INFO: Allocated block ID 159 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-159 2024-03-07 15:08:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-159 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-159 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 103 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 103 2024-03-07 15:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c0022f8ceb2ba93256c496998b5e129 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a00fee5b-f6bd-4e1b-b558-be184e33f027_isrForCrosstalkSources_1125933_13.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a00fee5b-f6bd-4e1b-b558-be184e33f027_isrForCrosstalkSources_1125933_13.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06aacfd39c6e7b764b980c86f4dc560d with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/6cb3a710-c700-436c-bcda-03fdb3a825a2_isrForCrosstalkSources_1125914_62.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/6cb3a710-c700-436c-bcda-03fdb3a825a2_isrForCrosstalkSources_1125914_62.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e4299f0adea0517df3ef060b495fd36 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/8b6fa6f6-f9a7-494e-8970-4fd0ab23f692_isrForCrosstalkSources_1125930_44.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/8b6fa6f6-f9a7-494e-8970-4fd0ab23f692_isrForCrosstalkSources_1125930_44.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e7a8675eef84b1193966d924d3d49e7 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/6bfda748-565c-47d1-ab8e-311a7db32289_isrForCrosstalkSources_1125943_44.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/6bfda748-565c-47d1-ab8e-311a7db32289_isrForCrosstalkSources_1125943_44.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29dbf18391153424232e6eec5f41eb1a with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/15a631c7-1465-448e-82e3-89524dc7dca1_isrForCrosstalkSources_1125892_58.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/15a631c7-1465-448e-82e3-89524dc7dca1_isrForCrosstalkSources_1125892_58.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb562bd80810ca44ccd85aca317978c6 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/1583bcf5-0154-43a8-8dc1-0cdd2e89c3d1_isrForCrosstalkSources_1125909_55.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/1583bcf5-0154-43a8-8dc1-0cdd2e89c3d1_isrForCrosstalkSources_1125909_55.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c4320c03db05174db318343dbe8b6a3 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/1e808b20-9309-4d2c-8692-cb9ed1e937d1_isrForCrosstalkSources_1125931_28.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/1e808b20-9309-4d2c-8692-cb9ed1e937d1_isrForCrosstalkSources_1125931_28.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c3d5428c0c0848658f0c42eedc59f8d with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/124634f3-e45c-41d9-a6c5-8b19dc37d26d_isrForCrosstalkSources_1125922_60.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/124634f3-e45c-41d9-a6c5-8b19dc37d26d_isrForCrosstalkSources_1125922_60.stderr 2024-03-07 15:08:31 proc_lsst.multi:170 [INFO] [multi] job_id 16896559 proc_lsst.multi INFO: [multi] job_id 16896559 2024-03-07 15:08:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 104 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 104 2024-03-07 15:08:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896559 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896559 parsl.executors.status_handling DEBUG: Launched block 159 on executor multi with job ID 16896559 parsl.executors.status_handling INFO: Allocated block ID 160 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-160 2024-03-07 15:08:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-160 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-160 2024-03-07 15:08:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 104 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 104 2024-03-07 15:08:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:31 proc_lsst.multi:170 [INFO] [multi] job_id 16896561 proc_lsst.multi INFO: [multi] job_id 16896561 2024-03-07 15:08:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 105 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 105 2024-03-07 15:08:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896561 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896561 parsl.executors.status_handling DEBUG: Launched block 160 on executor multi with job ID 16896561 parsl.executors.status_handling INFO: Allocated block ID 161 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-161 2024-03-07 15:08:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-161 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-161 2024-03-07 15:08:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 105 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 105 2024-03-07 15:08:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:32 proc_lsst.multi:170 [INFO] [multi] job_id 16896562 proc_lsst.multi INFO: [multi] job_id 16896562 2024-03-07 15:08:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 106 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 106 2024-03-07 15:08:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896562 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896562 parsl.executors.status_handling DEBUG: Launched block 161 on executor multi with job ID 16896562 parsl.executors.status_handling INFO: Allocated block ID 162 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-162 2024-03-07 15:08:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-162 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-162 2024-03-07 15:08:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 106 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 106 2024-03-07 15:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:32 proc_lsst.multi:170 [INFO] [multi] job_id 16896563 proc_lsst.multi INFO: [multi] job_id 16896563 2024-03-07 15:08:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 107 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 107 2024-03-07 15:08:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896563 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896563 parsl.executors.status_handling DEBUG: Launched block 162 on executor multi with job ID 16896563 parsl.executors.status_handling INFO: Allocated block ID 163 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-163 2024-03-07 15:08:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-163 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-163 2024-03-07 15:08:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 107 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 107 2024-03-07 15:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:32 proc_lsst.multi:170 [INFO] [multi] job_id 16896564 proc_lsst.multi INFO: [multi] job_id 16896564 2024-03-07 15:08:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 108 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 108 2024-03-07 15:08:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896564 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896564 parsl.executors.status_handling DEBUG: Launched block 163 on executor multi with job ID 16896564 parsl.executors.status_handling INFO: Allocated block ID 164 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-164 2024-03-07 15:08:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-164 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-164 2024-03-07 15:08:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 108 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 108 2024-03-07 15:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:33 proc_lsst.multi:170 [INFO] [multi] job_id 16896565 proc_lsst.multi INFO: [multi] job_id 16896565 2024-03-07 15:08:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 109 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 109 2024-03-07 15:08:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896565 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896565 parsl.executors.status_handling DEBUG: Launched block 164 on executor multi with job ID 16896565 parsl.executors.status_handling INFO: Allocated block ID 165 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-165 2024-03-07 15:08:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-165 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-165 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 109 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 109 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:33 proc_lsst.multi:170 [INFO] [multi] job_id 16896566 proc_lsst.multi INFO: [multi] job_id 16896566 2024-03-07 15:08:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 110 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 110 2024-03-07 15:08:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896566 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896566 parsl.executors.status_handling DEBUG: Launched block 165 on executor multi with job ID 16896566 parsl.executors.status_handling INFO: Allocated block ID 166 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-166 2024-03-07 15:08:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-166 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-166 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 110 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 110 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:33 proc_lsst.multi:170 [INFO] [multi] job_id 16896568 proc_lsst.multi INFO: [multi] job_id 16896568 2024-03-07 15:08:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 111 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 111 2024-03-07 15:08:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896568 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896568 parsl.executors.status_handling DEBUG: Launched block 166 on executor multi with job ID 16896568 parsl.executors.status_handling INFO: Allocated block ID 167 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-167 2024-03-07 15:08:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-167 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-167 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 111 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 111 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:33 proc_lsst.multi:170 [INFO] [multi] job_id 16896569 proc_lsst.multi INFO: [multi] job_id 16896569 2024-03-07 15:08:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 112 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 112 2024-03-07 15:08:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896569 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896569 parsl.executors.status_handling DEBUG: Launched block 167 on executor multi with job ID 16896569 parsl.executors.status_handling INFO: Allocated block ID 168 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-168 2024-03-07 15:08:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-168 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-168 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 112 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 112 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:33 proc_lsst.multi:170 [INFO] [multi] job_id 16896570 proc_lsst.multi INFO: [multi] job_id 16896570 2024-03-07 15:08:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 113 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 113 2024-03-07 15:08:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896570 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896570 parsl.executors.status_handling DEBUG: Launched block 168 on executor multi with job ID 16896570 parsl.executors.status_handling INFO: Allocated block ID 169 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-169 2024-03-07 15:08:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-169 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-169 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 113 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 113 2024-03-07 15:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:34 proc_lsst.multi:170 [INFO] [multi] job_id 16896571 proc_lsst.multi INFO: [multi] job_id 16896571 2024-03-07 15:08:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 114 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 114 2024-03-07 15:08:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896571 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896571 parsl.executors.status_handling DEBUG: Launched block 169 on executor multi with job ID 16896571 parsl.executors.status_handling INFO: Allocated block ID 170 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-170 2024-03-07 15:08:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-170 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-170 2024-03-07 15:08:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 114 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 114 2024-03-07 15:08:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:34 proc_lsst.multi:170 [INFO] [multi] job_id 16896572 proc_lsst.multi INFO: [multi] job_id 16896572 2024-03-07 15:08:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 115 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 115 2024-03-07 15:08:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896572 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896572 parsl.executors.status_handling DEBUG: Launched block 170 on executor multi with job ID 16896572 parsl.executors.status_handling INFO: Allocated block ID 171 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-171 2024-03-07 15:08:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-171 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-171 2024-03-07 15:08:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 115 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 115 2024-03-07 15:08:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:35 proc_lsst.multi:170 [INFO] [multi] job_id 16896573 proc_lsst.multi INFO: [multi] job_id 16896573 2024-03-07 15:08:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 116 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 116 2024-03-07 15:08:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896573 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896573 parsl.executors.status_handling DEBUG: Launched block 171 on executor multi with job ID 16896573 parsl.executors.status_handling INFO: Allocated block ID 172 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-172 2024-03-07 15:08:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-172 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-172 2024-03-07 15:08:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 116 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 116 2024-03-07 15:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:35 proc_lsst.multi:170 [INFO] [multi] job_id 16896574 proc_lsst.multi INFO: [multi] job_id 16896574 2024-03-07 15:08:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 117 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 117 2024-03-07 15:08:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896574 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896574 parsl.executors.status_handling DEBUG: Launched block 172 on executor multi with job ID 16896574 parsl.executors.status_handling INFO: Allocated block ID 173 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-173 2024-03-07 15:08:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-173 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-173 2024-03-07 15:08:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 117 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 117 2024-03-07 15:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:35 proc_lsst.multi:170 [INFO] [multi] job_id 16896575 proc_lsst.multi INFO: [multi] job_id 16896575 2024-03-07 15:08:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 118 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 118 2024-03-07 15:08:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896575 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896575 parsl.executors.status_handling DEBUG: Launched block 173 on executor multi with job ID 16896575 parsl.executors.status_handling INFO: Allocated block ID 174 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-174 2024-03-07 15:08:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-174 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-174 2024-03-07 15:08:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 118 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 118 2024-03-07 15:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 769c7d6a49d43c49c8d2b6e6ba99d715 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/39d8228e-c0b9-4b03-b8bc-9662cf7f4930_isrForCrosstalkSources_1125914_46.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/39d8228e-c0b9-4b03-b8bc-9662cf7f4930_isrForCrosstalkSources_1125914_46.stderr 2024-03-07 15:08:36 proc_lsst.multi:170 [INFO] [multi] job_id 16896576 proc_lsst.multi INFO: [multi] job_id 16896576 2024-03-07 15:08:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 119 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 119 2024-03-07 15:08:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896576 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896576 parsl.executors.status_handling DEBUG: Launched block 174 on executor multi with job ID 16896576 parsl.executors.status_handling INFO: Allocated block ID 175 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-175 2024-03-07 15:08:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-175 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-175 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 119 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 119 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3dc60a003d597f44b28b41ce20d441f with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/0b1ab236-65c2-4eff-82c6-9d82aaf42945_isrForCrosstalkSources_1125901_21.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/0b1ab236-65c2-4eff-82c6-9d82aaf42945_isrForCrosstalkSources_1125901_21.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c40794c2025df8cf7d0805507fa669df with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/875dc908-4886-49cc-8c11-f65a0ce412c4_isrForCrosstalkSources_1125943_45.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/875dc908-4886-49cc-8c11-f65a0ce412c4_isrForCrosstalkSources_1125943_45.stderr 2024-03-07 15:08:36 proc_lsst.multi:170 [INFO] [multi] job_id 16896577 proc_lsst.multi INFO: [multi] job_id 16896577 2024-03-07 15:08:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 120 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 120 2024-03-07 15:08:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896577 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896577 parsl.executors.status_handling DEBUG: Launched block 175 on executor multi with job ID 16896577 parsl.executors.status_handling INFO: Allocated block ID 176 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-176 2024-03-07 15:08:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-176 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-176 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 120 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 120 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:36 proc_lsst.multi:170 [INFO] [multi] job_id 16896578 proc_lsst.multi INFO: [multi] job_id 16896578 2024-03-07 15:08:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 121 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 121 2024-03-07 15:08:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896578 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896578 parsl.executors.status_handling DEBUG: Launched block 176 on executor multi with job ID 16896578 parsl.executors.status_handling INFO: Allocated block ID 177 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-177 2024-03-07 15:08:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-177 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-177 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 121 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 121 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:36 proc_lsst.multi:170 [INFO] [multi] job_id 16896579 proc_lsst.multi INFO: [multi] job_id 16896579 2024-03-07 15:08:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 122 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 122 2024-03-07 15:08:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896579 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896579 parsl.executors.status_handling DEBUG: Launched block 177 on executor multi with job ID 16896579 parsl.executors.status_handling INFO: Allocated block ID 178 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-178 2024-03-07 15:08:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-178 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-178 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 122 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 122 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df923e0573b85cd0b03e3059a1886454 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/5ca828bc-9f09-481b-acfb-b8c3cc97c050_isrForCrosstalkSources_1125889_6.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/5ca828bc-9f09-481b-acfb-b8c3cc97c050_isrForCrosstalkSources_1125889_6.stderr 2024-03-07 15:08:36 proc_lsst.multi:170 [INFO] [multi] job_id 16896580 proc_lsst.multi INFO: [multi] job_id 16896580 2024-03-07 15:08:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 123 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 123 2024-03-07 15:08:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896580 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896580 parsl.executors.status_handling DEBUG: Launched block 178 on executor multi with job ID 16896580 parsl.executors.status_handling INFO: Allocated block ID 179 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-179 2024-03-07 15:08:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-179 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-179 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 123 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 123 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:36 proc_lsst.multi:170 [INFO] [multi] job_id 16896581 proc_lsst.multi INFO: [multi] job_id 16896581 2024-03-07 15:08:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 124 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 124 2024-03-07 15:08:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896581 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896581 parsl.executors.status_handling DEBUG: Launched block 179 on executor multi with job ID 16896581 parsl.executors.status_handling INFO: Allocated block ID 180 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-180 2024-03-07 15:08:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-180 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-180 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 124 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 124 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8c96e2eb933f11cb0718d50c280df80 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/fbcde238-809a-45a0-8621-5a4ac58c541f_isrForCrosstalkSources_1125933_56.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/fbcde238-809a-45a0-8621-5a4ac58c541f_isrForCrosstalkSources_1125933_56.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4223c0ad26244da6f022482c90f9da9c with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d0c61ff1-1178-4933-b89a-fa78eb187790_isrForCrosstalkSources_1125890_5.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d0c61ff1-1178-4933-b89a-fa78eb187790_isrForCrosstalkSources_1125890_5.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28e9a0ae20b55c2a01cc49826363f6fd with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/d368dd3a-b708-44da-92c6-9152bba4da44_isrForCrosstalkSources_1125921_15.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/d368dd3a-b708-44da-92c6-9152bba4da44_isrForCrosstalkSources_1125921_15.stderr 2024-03-07 15:08:36 proc_lsst.multi:170 [INFO] [multi] job_id 16896582 proc_lsst.multi INFO: [multi] job_id 16896582 2024-03-07 15:08:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 125 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 125 2024-03-07 15:08:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896582 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896582 parsl.executors.status_handling DEBUG: Launched block 180 on executor multi with job ID 16896582 parsl.executors.status_handling INFO: Allocated block ID 181 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-181 2024-03-07 15:08:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-181 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-181 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 125 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 125 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:36 proc_lsst.multi:170 [INFO] [multi] job_id 16896583 proc_lsst.multi INFO: [multi] job_id 16896583 2024-03-07 15:08:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 126 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 126 2024-03-07 15:08:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896583 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896583 parsl.executors.status_handling DEBUG: Launched block 181 on executor multi with job ID 16896583 parsl.executors.status_handling INFO: Allocated block ID 182 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-182 2024-03-07 15:08:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-182 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-182 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 126 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 126 2024-03-07 15:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 288ad8ef1d551e08cac0904e565f2324 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/1dc8ab4b-67ec-467a-bf05-701dbbd60d8a_isrForCrosstalkSources_1125919_8.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/1dc8ab4b-67ec-467a-bf05-701dbbd60d8a_isrForCrosstalkSources_1125919_8.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ad455f41a9d5f4b1ef64d5b5fc25486 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/6b81133c-8466-42fa-9a32-4b29b050696e_isrForCrosstalkSources_1125933_42.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/6b81133c-8466-42fa-9a32-4b29b050696e_isrForCrosstalkSources_1125933_42.stderr 2024-03-07 15:08:37 proc_lsst.multi:170 [INFO] [multi] job_id 16896584 proc_lsst.multi INFO: [multi] job_id 16896584 2024-03-07 15:08:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 127 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 127 2024-03-07 15:08:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896584 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896584 parsl.executors.status_handling DEBUG: Launched block 182 on executor multi with job ID 16896584 parsl.executors.status_handling INFO: Allocated block ID 183 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-183 2024-03-07 15:08:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-183 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-183 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 127 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 127 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18ab91bc9ba24e7940eb144a4d29d506 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/104abc40-de73-42e1-b0a3-dac873cf2a57_isrForCrosstalkSources_1125912_3.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/104abc40-de73-42e1-b0a3-dac873cf2a57_isrForCrosstalkSources_1125912_3.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a05788ece13226fd5a9b7bb000b98ca with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/9a92e92c-c071-48ef-940d-b01f81fb19c1_isrForCrosstalkSources_1125904_41.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/9a92e92c-c071-48ef-940d-b01f81fb19c1_isrForCrosstalkSources_1125904_41.stderr 2024-03-07 15:08:37 proc_lsst.multi:170 [INFO] [multi] job_id 16896585 proc_lsst.multi INFO: [multi] job_id 16896585 2024-03-07 15:08:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 128 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 128 2024-03-07 15:08:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896585 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896585 parsl.executors.status_handling DEBUG: Launched block 183 on executor multi with job ID 16896585 parsl.executors.status_handling INFO: Allocated block ID 184 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-184 2024-03-07 15:08:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-184 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-184 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 128 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 128 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:37 proc_lsst.multi:170 [INFO] [multi] job_id 16896586 proc_lsst.multi INFO: [multi] job_id 16896586 2024-03-07 15:08:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 129 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 129 2024-03-07 15:08:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896586 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896586 parsl.executors.status_handling DEBUG: Launched block 184 on executor multi with job ID 16896586 parsl.executors.status_handling INFO: Allocated block ID 185 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-185 2024-03-07 15:08:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-185 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-185 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 129 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 129 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47606efb6e1eace9713535930c6377fb with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/e08ceb83-43d3-4616-b4ea-29cfd398b063_isrForCrosstalkSources_1125899_7.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/e08ceb83-43d3-4616-b4ea-29cfd398b063_isrForCrosstalkSources_1125899_7.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d5c65a56eec662bac22b231f67a2750 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/d866a3e5-3e5f-41a4-b297-a6d254cb1401_isrForCrosstalkSources_1125932_5.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/d866a3e5-3e5f-41a4-b297-a6d254cb1401_isrForCrosstalkSources_1125932_5.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cda6290eef5f90f450be78ab2c0941fa with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/5a5aa490-fda3-407d-a3cb-d711e6ccfd60_isrForCrosstalkSources_1125913_61.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/5a5aa490-fda3-407d-a3cb-d711e6ccfd60_isrForCrosstalkSources_1125913_61.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21abda7800122fd7c02fd6692fe2b988 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/0d1026d4-baa3-47dd-aea6-5ca7ad490272_isrForCrosstalkSources_1125905_26.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/0d1026d4-baa3-47dd-aea6-5ca7ad490272_isrForCrosstalkSources_1125905_26.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 097d823d3d18de31729cbd8f45cf0415 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/3bf68d9a-906b-48c1-bd5a-e9fa5760fb2b_isrForCrosstalkSources_1125904_55.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/3bf68d9a-906b-48c1-bd5a-e9fa5760fb2b_isrForCrosstalkSources_1125904_55.stderr 2024-03-07 15:08:37 proc_lsst.multi:170 [INFO] [multi] job_id 16896587 proc_lsst.multi INFO: [multi] job_id 16896587 2024-03-07 15:08:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 130 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 130 2024-03-07 15:08:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896587 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896587 parsl.executors.status_handling DEBUG: Launched block 185 on executor multi with job ID 16896587 parsl.executors.status_handling INFO: Allocated block ID 186 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-186 2024-03-07 15:08:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-186 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-186 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 130 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 130 2024-03-07 15:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4148fc4f441c0a7ca705094a1fb431d6 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/fd818e34-93e3-471f-9131-414591e28826_isrForCrosstalkSources_1125942_31.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/fd818e34-93e3-471f-9131-414591e28826_isrForCrosstalkSources_1125942_31.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f99b89fe1e9ffe77d2a20b1c9abd8d74 with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/32e622d1-8143-44d2-a017-337194921ac9_isrForCrosstalkSources_1125894_20.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/32e622d1-8143-44d2-a017-337194921ac9_isrForCrosstalkSources_1125894_20.stderr 2024-03-07 15:08:38 proc_lsst.multi:170 [INFO] [multi] job_id 16896588 proc_lsst.multi INFO: [multi] job_id 16896588 2024-03-07 15:08:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 131 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 131 2024-03-07 15:08:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896588 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896588 parsl.executors.status_handling DEBUG: Launched block 186 on executor multi with job ID 16896588 parsl.executors.status_handling INFO: Allocated block ID 187 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-187 2024-03-07 15:08:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-187 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-187 2024-03-07 15:08:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 131 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 131 2024-03-07 15:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6e4cc18589317350136f91cdc754783 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/40e989d0-99fe-4630-8403-9c15f4fdefb3_isrForCrosstalkSources_1125907_52.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/40e989d0-99fe-4630-8403-9c15f4fdefb3_isrForCrosstalkSources_1125907_52.stderr 2024-03-07 15:08:38 proc_lsst.multi:170 [INFO] [multi] job_id 16896589 proc_lsst.multi INFO: [multi] job_id 16896589 2024-03-07 15:08:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 132 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 132 2024-03-07 15:08:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896589 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896589 parsl.executors.status_handling DEBUG: Launched block 187 on executor multi with job ID 16896589 parsl.executors.status_handling INFO: Allocated block ID 188 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-188 2024-03-07 15:08:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-188 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-188 2024-03-07 15:08:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 132 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 132 2024-03-07 15:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38e34be71947fc8a7ed948e905b55d8a with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/2f36088b-2ec8-49a7-bb19-b0cb577b9427_isrForCrosstalkSources_1125929_50.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/2f36088b-2ec8-49a7-bb19-b0cb577b9427_isrForCrosstalkSources_1125929_50.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a148013c205d6c2281dfd60b906d7887 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/8b6c58e7-aa70-427b-9ac2-5b411faf0d5e_isrForCrosstalkSources_1125924_54.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/8b6c58e7-aa70-427b-9ac2-5b411faf0d5e_isrForCrosstalkSources_1125924_54.stderr 2024-03-07 15:08:38 proc_lsst.multi:170 [INFO] [multi] job_id 16896590 proc_lsst.multi INFO: [multi] job_id 16896590 2024-03-07 15:08:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 133 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 133 2024-03-07 15:08:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896590 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896590 parsl.executors.status_handling DEBUG: Launched block 188 on executor multi with job ID 16896590 parsl.executors.status_handling INFO: Allocated block ID 189 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-189 2024-03-07 15:08:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-189 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-189 2024-03-07 15:08:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 133 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 133 2024-03-07 15:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:39 proc_lsst.multi:170 [INFO] [multi] job_id 16896591 proc_lsst.multi INFO: [multi] job_id 16896591 2024-03-07 15:08:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 134 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 134 2024-03-07 15:08:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896591 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896591 parsl.executors.status_handling DEBUG: Launched block 189 on executor multi with job ID 16896591 parsl.executors.status_handling INFO: Allocated block ID 190 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-190 2024-03-07 15:08:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-190 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-190 2024-03-07 15:08:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 134 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 134 2024-03-07 15:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:39 proc_lsst.multi:170 [INFO] [multi] job_id 16896592 proc_lsst.multi INFO: [multi] job_id 16896592 2024-03-07 15:08:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 135 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 135 2024-03-07 15:08:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896592 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896592 parsl.executors.status_handling DEBUG: Launched block 190 on executor multi with job ID 16896592 parsl.executors.status_handling INFO: Allocated block ID 191 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-191 2024-03-07 15:08:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-191 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-191 2024-03-07 15:08:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 135 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 135 2024-03-07 15:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:39 proc_lsst.multi:170 [INFO] [multi] job_id 16896593 proc_lsst.multi INFO: [multi] job_id 16896593 2024-03-07 15:08:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 136 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 136 2024-03-07 15:08:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896593 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896593 parsl.executors.status_handling DEBUG: Launched block 191 on executor multi with job ID 16896593 parsl.executors.status_handling INFO: Allocated block ID 192 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-192 2024-03-07 15:08:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-192 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-192 2024-03-07 15:08:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 136 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 136 2024-03-07 15:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:40 proc_lsst.multi:170 [INFO] [multi] job_id 16896594 proc_lsst.multi INFO: [multi] job_id 16896594 2024-03-07 15:08:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 137 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 137 2024-03-07 15:08:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896594 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896594 parsl.executors.status_handling DEBUG: Launched block 192 on executor multi with job ID 16896594 parsl.executors.status_handling INFO: Allocated block ID 193 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-193 2024-03-07 15:08:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-193 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-193 2024-03-07 15:08:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 137 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 137 2024-03-07 15:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:40 proc_lsst.multi:170 [INFO] [multi] job_id 16896595 proc_lsst.multi INFO: [multi] job_id 16896595 2024-03-07 15:08:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 138 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 138 2024-03-07 15:08:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896595 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896595 parsl.executors.status_handling DEBUG: Launched block 193 on executor multi with job ID 16896595 parsl.executors.status_handling INFO: Allocated block ID 194 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-194 2024-03-07 15:08:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-194 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-194 2024-03-07 15:08:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 138 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 138 2024-03-07 15:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:40 proc_lsst.multi:170 [INFO] [multi] job_id 16896596 proc_lsst.multi INFO: [multi] job_id 16896596 2024-03-07 15:08:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 139 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 139 2024-03-07 15:08:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896596 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896596 parsl.executors.status_handling DEBUG: Launched block 194 on executor multi with job ID 16896596 parsl.executors.status_handling INFO: Allocated block ID 195 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-195 2024-03-07 15:08:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-195 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-195 2024-03-07 15:08:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 139 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 139 2024-03-07 15:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e653b14f28673e9f39c2d92661173bf with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/0a74a70b-a270-4de1-b515-924cdeeb1277_isrForCrosstalkSources_1125896_19.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/0a74a70b-a270-4de1-b515-924cdeeb1277_isrForCrosstalkSources_1125896_19.stderr 2024-03-07 15:08:41 proc_lsst.multi:170 [INFO] [multi] job_id 16896597 proc_lsst.multi INFO: [multi] job_id 16896597 2024-03-07 15:08:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 140 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 140 2024-03-07 15:08:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896597 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896597 parsl.executors.status_handling DEBUG: Launched block 195 on executor multi with job ID 16896597 parsl.executors.status_handling INFO: Allocated block ID 196 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-196 2024-03-07 15:08:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-196 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-196 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 140 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 140 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:41 proc_lsst.multi:170 [INFO] [multi] job_id 16896598 proc_lsst.multi INFO: [multi] job_id 16896598 2024-03-07 15:08:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 141 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 141 2024-03-07 15:08:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896598 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896598 parsl.executors.status_handling DEBUG: Launched block 196 on executor multi with job ID 16896598 parsl.executors.status_handling INFO: Allocated block ID 197 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-197 2024-03-07 15:08:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-197 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-197 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 141 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 141 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acef1eb7132e322670788340bc1f0ae3 with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/168d01f6-b0c2-443c-b581-33fd5372dfa2_isrForCrosstalkSources_1125936_24.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/168d01f6-b0c2-443c-b581-33fd5372dfa2_isrForCrosstalkSources_1125936_24.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ccf345eedd7b072d3c369d9557e7fa6 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ce9121f1-c70a-4baa-b618-e4ef46cce3ee_isrForCrosstalkSources_1125947_23.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ce9121f1-c70a-4baa-b618-e4ef46cce3ee_isrForCrosstalkSources_1125947_23.stderr 2024-03-07 15:08:41 proc_lsst.multi:170 [INFO] [multi] job_id 16896599 proc_lsst.multi INFO: [multi] job_id 16896599 2024-03-07 15:08:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 142 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 142 2024-03-07 15:08:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896599 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896599 parsl.executors.status_handling DEBUG: Launched block 197 on executor multi with job ID 16896599 parsl.executors.status_handling INFO: Allocated block ID 198 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-198 2024-03-07 15:08:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-198 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-198 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 142 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 142 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:41 proc_lsst.multi:170 [INFO] [multi] job_id 16896600 proc_lsst.multi INFO: [multi] job_id 16896600 2024-03-07 15:08:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 143 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 143 2024-03-07 15:08:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896600 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896600 parsl.executors.status_handling DEBUG: Launched block 198 on executor multi with job ID 16896600 parsl.executors.status_handling INFO: Allocated block ID 199 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-199 2024-03-07 15:08:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-199 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-199 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 143 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 143 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:41 proc_lsst.multi:170 [INFO] [multi] job_id 16896601 proc_lsst.multi INFO: [multi] job_id 16896601 2024-03-07 15:08:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 144 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 144 2024-03-07 15:08:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896601 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896601 parsl.executors.status_handling DEBUG: Launched block 199 on executor multi with job ID 16896601 parsl.executors.status_handling INFO: Allocated block ID 200 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-200 2024-03-07 15:08:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-200 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-200 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 144 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 144 2024-03-07 15:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4e064553c08d287b3534d3cd41f4ed0 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/e0026bb8-9de7-4116-a0cd-149ec7d1b8a4_isrForCrosstalkSources_1125915_35.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/e0026bb8-9de7-4116-a0cd-149ec7d1b8a4_isrForCrosstalkSources_1125915_35.stderr 2024-03-07 15:08:42 proc_lsst.multi:170 [INFO] [multi] job_id 16896602 proc_lsst.multi INFO: [multi] job_id 16896602 2024-03-07 15:08:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 145 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 145 2024-03-07 15:08:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896602 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896602 parsl.executors.status_handling DEBUG: Launched block 200 on executor multi with job ID 16896602 parsl.executors.status_handling INFO: Allocated block ID 201 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-201 2024-03-07 15:08:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-201 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-201 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 145 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 145 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ecf36e81f8f914e6af66f8c4b1253c9 with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/8fc8d46f-5578-482d-8ecf-669137467ffe_isrForCrosstalkSources_1125942_48.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/8fc8d46f-5578-482d-8ecf-669137467ffe_isrForCrosstalkSources_1125942_48.stderr 2024-03-07 15:08:42 proc_lsst.multi:170 [INFO] [multi] job_id 16896603 proc_lsst.multi INFO: [multi] job_id 16896603 2024-03-07 15:08:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 146 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 146 2024-03-07 15:08:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896603 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896603 parsl.executors.status_handling DEBUG: Launched block 201 on executor multi with job ID 16896603 parsl.executors.status_handling INFO: Allocated block ID 202 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-202 2024-03-07 15:08:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-202 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-202 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 146 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 146 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e07de5b1e320785b823fa50427f0329 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/6efdc2d4-d6d5-4712-b45a-3b37b20a68d9_isrForCrosstalkSources_1125914_13.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/6efdc2d4-d6d5-4712-b45a-3b37b20a68d9_isrForCrosstalkSources_1125914_13.stderr 2024-03-07 15:08:42 proc_lsst.multi:170 [INFO] [multi] job_id 16896604 proc_lsst.multi INFO: [multi] job_id 16896604 2024-03-07 15:08:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 147 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 147 2024-03-07 15:08:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896604 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896604 parsl.executors.status_handling DEBUG: Launched block 202 on executor multi with job ID 16896604 parsl.executors.status_handling INFO: Allocated block ID 203 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-203 2024-03-07 15:08:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-203 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-203 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 147 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 147 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:42 proc_lsst.multi:170 [INFO] [multi] job_id 16896605 proc_lsst.multi INFO: [multi] job_id 16896605 2024-03-07 15:08:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 148 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 148 2024-03-07 15:08:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896605 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896605 parsl.executors.status_handling DEBUG: Launched block 203 on executor multi with job ID 16896605 parsl.executors.status_handling INFO: Allocated block ID 204 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-204 2024-03-07 15:08:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-204 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-204 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 148 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 148 2024-03-07 15:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a68a26a8f4f8908025a09669ba99e26f with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/5325bf4b-4b7b-4056-919b-86decb3f5ec3_isrForCrosstalkSources_1125900_8.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/5325bf4b-4b7b-4056-919b-86decb3f5ec3_isrForCrosstalkSources_1125900_8.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c505b7a7fdacb5ab1eb2041a91c28f38 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/31c73926-f8e4-4505-92d3-987c7715594d_isrForCrosstalkSources_1125901_39.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/31c73926-f8e4-4505-92d3-987c7715594d_isrForCrosstalkSources_1125901_39.stderr 2024-03-07 15:08:43 proc_lsst.multi:170 [INFO] [multi] job_id 16896606 proc_lsst.multi INFO: [multi] job_id 16896606 2024-03-07 15:08:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 149 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 149 2024-03-07 15:08:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896606 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896606 parsl.executors.status_handling DEBUG: Launched block 204 on executor multi with job ID 16896606 parsl.executors.status_handling INFO: Allocated block ID 205 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-205 2024-03-07 15:08:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-205 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-205 2024-03-07 15:08:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 149 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 149 2024-03-07 15:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50070c9f97659c0c33872bf52cf1032c with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/d002acac-1111-4e9a-83fd-70109ad90c7c_isrForCrosstalkSources_1125939_56.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/d002acac-1111-4e9a-83fd-70109ad90c7c_isrForCrosstalkSources_1125939_56.stderr 2024-03-07 15:08:43 proc_lsst.multi:170 [INFO] [multi] job_id 16896607 proc_lsst.multi INFO: [multi] job_id 16896607 2024-03-07 15:08:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 150 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 150 2024-03-07 15:08:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896607 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896607 parsl.executors.status_handling DEBUG: Launched block 205 on executor multi with job ID 16896607 parsl.executors.status_handling INFO: Allocated block ID 206 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-206 2024-03-07 15:08:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-206 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-206 2024-03-07 15:08:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 150 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 150 2024-03-07 15:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5984b698cef65dce95a59a2b283d0f72 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/125d83bb-feb4-400b-ad61-3ba5f27aa121_isrForCrosstalkSources_1125908_49.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/125d83bb-feb4-400b-ad61-3ba5f27aa121_isrForCrosstalkSources_1125908_49.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea4938510a31bc854adff79b9ca3e5c5 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/a26318d6-e318-472f-950a-85d1c75067df_isrForCrosstalkSources_1125944_30.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/a26318d6-e318-472f-950a-85d1c75067df_isrForCrosstalkSources_1125944_30.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa9b44a7df12b6135ef81b0f9c17671c with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/e32e9e8b-e0d6-48cd-a646-968e6a6f9ad9_isrForCrosstalkSources_1125919_36.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/e32e9e8b-e0d6-48cd-a646-968e6a6f9ad9_isrForCrosstalkSources_1125919_36.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43057276427e5a3da133100b58a41dab with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/94552628-df7b-473d-9d6c-48934f87876c_isrForCrosstalkSources_1125908_61.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/94552628-df7b-473d-9d6c-48934f87876c_isrForCrosstalkSources_1125908_61.stderr 2024-03-07 15:08:43 proc_lsst.multi:170 [INFO] [multi] job_id 16896608 proc_lsst.multi INFO: [multi] job_id 16896608 2024-03-07 15:08:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 151 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 151 2024-03-07 15:08:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896608 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896608 parsl.executors.status_handling DEBUG: Launched block 206 on executor multi with job ID 16896608 parsl.executors.status_handling INFO: Allocated block ID 207 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-207 2024-03-07 15:08:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-207 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-207 2024-03-07 15:08:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 151 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 151 2024-03-07 15:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:44 proc_lsst.multi:170 [INFO] [multi] job_id 16896609 proc_lsst.multi INFO: [multi] job_id 16896609 2024-03-07 15:08:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 152 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 152 2024-03-07 15:08:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896609 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896609 parsl.executors.status_handling DEBUG: Launched block 207 on executor multi with job ID 16896609 parsl.executors.status_handling INFO: Allocated block ID 208 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-208 2024-03-07 15:08:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-208 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-208 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 152 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 152 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:44 proc_lsst.multi:170 [INFO] [multi] job_id 16896610 proc_lsst.multi INFO: [multi] job_id 16896610 2024-03-07 15:08:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 153 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 153 2024-03-07 15:08:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896610 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896610 parsl.executors.status_handling DEBUG: Launched block 208 on executor multi with job ID 16896610 parsl.executors.status_handling INFO: Allocated block ID 209 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-209 2024-03-07 15:08:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-209 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-209 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 153 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 153 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:44 proc_lsst.multi:170 [INFO] [multi] job_id 16896611 proc_lsst.multi INFO: [multi] job_id 16896611 2024-03-07 15:08:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 154 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 154 2024-03-07 15:08:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896611 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896611 parsl.executors.status_handling DEBUG: Launched block 209 on executor multi with job ID 16896611 parsl.executors.status_handling INFO: Allocated block ID 210 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-210 2024-03-07 15:08:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-210 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-210 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 154 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 154 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 915430bbfc431ef08b2d43579623c142 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/f02454e9-5731-4b7a-b55e-f14a7e6b0568_isrForCrosstalkSources_1125938_58.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/f02454e9-5731-4b7a-b55e-f14a7e6b0568_isrForCrosstalkSources_1125938_58.stderr 2024-03-07 15:08:44 proc_lsst.multi:170 [INFO] [multi] job_id 16896612 proc_lsst.multi INFO: [multi] job_id 16896612 2024-03-07 15:08:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 155 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 155 2024-03-07 15:08:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896612 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896612 parsl.executors.status_handling DEBUG: Launched block 210 on executor multi with job ID 16896612 parsl.executors.status_handling INFO: Allocated block ID 211 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-211 2024-03-07 15:08:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-211 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-211 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 155 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 155 2024-03-07 15:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:45 proc_lsst.multi:170 [INFO] [multi] job_id 16896613 proc_lsst.multi INFO: [multi] job_id 16896613 2024-03-07 15:08:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 156 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 156 2024-03-07 15:08:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:45 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896613 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896613 parsl.executors.status_handling DEBUG: Launched block 211 on executor multi with job ID 16896613 parsl.executors.status_handling INFO: Allocated block ID 212 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-212 2024-03-07 15:08:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-212 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-212 2024-03-07 15:08:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:45 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 156 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 156 2024-03-07 15:08:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:45 proc_lsst.multi:170 [INFO] [multi] job_id 16896614 proc_lsst.multi INFO: [multi] job_id 16896614 2024-03-07 15:08:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 157 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 157 2024-03-07 15:08:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:45 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896614 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896614 parsl.executors.status_handling DEBUG: Launched block 212 on executor multi with job ID 16896614 parsl.executors.status_handling INFO: Allocated block ID 213 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-213 2024-03-07 15:08:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-213 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-213 2024-03-07 15:08:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:45 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 157 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 157 2024-03-07 15:08:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b7684d6a6d4c27bbf1c98675cb7b888 with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/2efeb706-cafd-4482-bceb-3e1bb6575d83_isrForCrosstalkSources_1125941_28.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/2efeb706-cafd-4482-bceb-3e1bb6575d83_isrForCrosstalkSources_1125941_28.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd2e80c37e90236e951edf3e88840075 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/6b286417-799c-425e-8dfe-c58530c18e92_isrForCrosstalkSources_1125911_5.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/6b286417-799c-425e-8dfe-c58530c18e92_isrForCrosstalkSources_1125911_5.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db2820f239905e7a52bbe76079b7a154 with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/4ecee11a-47d1-400d-b98f-a074d0c1c127_isrForCrosstalkSources_1125927_1.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/4ecee11a-47d1-400d-b98f-a074d0c1c127_isrForCrosstalkSources_1125927_1.stderr 2024-03-07 15:08:45 proc_lsst.multi:170 [INFO] [multi] job_id 16896615 proc_lsst.multi INFO: [multi] job_id 16896615 2024-03-07 15:08:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 158 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 158 2024-03-07 15:08:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:45 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896615 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896615 parsl.executors.status_handling DEBUG: Launched block 213 on executor multi with job ID 16896615 parsl.executors.status_handling INFO: Allocated block ID 214 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-214 2024-03-07 15:08:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-214 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-214 2024-03-07 15:08:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:45 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 158 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 158 2024-03-07 15:08:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:46 proc_lsst.multi:170 [INFO] [multi] job_id 16896616 proc_lsst.multi INFO: [multi] job_id 16896616 2024-03-07 15:08:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 159 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 159 2024-03-07 15:08:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896616 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896616 parsl.executors.status_handling DEBUG: Launched block 214 on executor multi with job ID 16896616 parsl.executors.status_handling INFO: Allocated block ID 215 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-215 2024-03-07 15:08:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-215 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-215 2024-03-07 15:08:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 159 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 159 2024-03-07 15:08:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:46 proc_lsst.multi:170 [INFO] [multi] job_id 16896617 proc_lsst.multi INFO: [multi] job_id 16896617 2024-03-07 15:08:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 160 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 160 2024-03-07 15:08:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896617 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896617 parsl.executors.status_handling DEBUG: Launched block 215 on executor multi with job ID 16896617 parsl.executors.status_handling INFO: Allocated block ID 216 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-216 2024-03-07 15:08:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-216 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-216 2024-03-07 15:08:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 160 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 160 2024-03-07 15:08:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:46 proc_lsst.multi:170 [INFO] [multi] job_id 16896618 proc_lsst.multi INFO: [multi] job_id 16896618 2024-03-07 15:08:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 161 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 161 2024-03-07 15:08:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896618 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896618 parsl.executors.status_handling DEBUG: Launched block 216 on executor multi with job ID 16896618 parsl.executors.status_handling INFO: Allocated block ID 217 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-217 2024-03-07 15:08:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-217 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-217 2024-03-07 15:08:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 161 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 161 2024-03-07 15:08:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:47 proc_lsst.multi:170 [INFO] [multi] job_id 16896619 proc_lsst.multi INFO: [multi] job_id 16896619 2024-03-07 15:08:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 162 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 162 2024-03-07 15:08:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896619 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896619 parsl.executors.status_handling DEBUG: Launched block 217 on executor multi with job ID 16896619 parsl.executors.status_handling INFO: Allocated block ID 218 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-218 2024-03-07 15:08:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-218 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-218 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 162 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 162 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:47 proc_lsst.multi:170 [INFO] [multi] job_id 16896620 proc_lsst.multi INFO: [multi] job_id 16896620 2024-03-07 15:08:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 163 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 163 2024-03-07 15:08:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896620 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896620 parsl.executors.status_handling DEBUG: Launched block 218 on executor multi with job ID 16896620 parsl.executors.status_handling INFO: Allocated block ID 219 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-219 2024-03-07 15:08:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-219 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-219 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 163 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 163 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:47 proc_lsst.multi:170 [INFO] [multi] job_id 16896621 proc_lsst.multi INFO: [multi] job_id 16896621 2024-03-07 15:08:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 164 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 164 2024-03-07 15:08:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896621 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896621 parsl.executors.status_handling DEBUG: Launched block 219 on executor multi with job ID 16896621 parsl.executors.status_handling INFO: Allocated block ID 220 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-220 2024-03-07 15:08:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-220 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-220 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 164 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 164 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:47 proc_lsst.multi:170 [INFO] [multi] job_id 16896622 proc_lsst.multi INFO: [multi] job_id 16896622 2024-03-07 15:08:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 165 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 165 2024-03-07 15:08:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896622 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896622 parsl.executors.status_handling DEBUG: Launched block 220 on executor multi with job ID 16896622 parsl.executors.status_handling INFO: Allocated block ID 221 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-221 2024-03-07 15:08:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-221 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-221 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 165 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 165 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:47 proc_lsst.multi:170 [INFO] [multi] job_id 16896623 proc_lsst.multi INFO: [multi] job_id 16896623 2024-03-07 15:08:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 166 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 166 2024-03-07 15:08:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896623 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896623 parsl.executors.status_handling DEBUG: Launched block 221 on executor multi with job ID 16896623 parsl.executors.status_handling INFO: Allocated block ID 222 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-222 2024-03-07 15:08:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-222 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-222 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 166 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 166 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:47 proc_lsst.multi:170 [INFO] [multi] job_id 16896624 proc_lsst.multi INFO: [multi] job_id 16896624 2024-03-07 15:08:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 167 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 167 2024-03-07 15:08:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896624 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896624 parsl.executors.status_handling DEBUG: Launched block 222 on executor multi with job ID 16896624 parsl.executors.status_handling INFO: Allocated block ID 223 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-223 2024-03-07 15:08:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-223 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-223 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 167 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 167 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:47 proc_lsst.multi:170 [INFO] [multi] job_id 16896625 proc_lsst.multi INFO: [multi] job_id 16896625 2024-03-07 15:08:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 168 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 168 2024-03-07 15:08:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896625 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896625 parsl.executors.status_handling DEBUG: Launched block 223 on executor multi with job ID 16896625 parsl.executors.status_handling INFO: Allocated block ID 224 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-224 2024-03-07 15:08:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-224 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-224 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 168 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 168 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:47 proc_lsst.multi:170 [INFO] [multi] job_id 16896626 proc_lsst.multi INFO: [multi] job_id 16896626 2024-03-07 15:08:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 169 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 169 2024-03-07 15:08:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896626 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896626 parsl.executors.status_handling DEBUG: Launched block 224 on executor multi with job ID 16896626 parsl.executors.status_handling INFO: Allocated block ID 225 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-225 2024-03-07 15:08:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-225 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-225 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 169 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 169 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:47 proc_lsst.multi:170 [INFO] [multi] job_id 16896627 proc_lsst.multi INFO: [multi] job_id 16896627 2024-03-07 15:08:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 170 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 170 2024-03-07 15:08:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896627 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896627 parsl.executors.status_handling DEBUG: Launched block 225 on executor multi with job ID 16896627 parsl.executors.status_handling INFO: Allocated block ID 226 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-226 2024-03-07 15:08:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-226 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-226 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 170 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 170 2024-03-07 15:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b277d0d3dd966900616dc695fbab8457 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/7c7c2bef-27e7-4ae9-b119-5f98351aaa34_isrForCrosstalkSources_1125943_39.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/7c7c2bef-27e7-4ae9-b119-5f98351aaa34_isrForCrosstalkSources_1125943_39.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1d19d35d965f6f81246451ceb1d0fba with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b0690202-afa1-4e98-9d00-48f256240cc9_isrForCrosstalkSources_1125934_51.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b0690202-afa1-4e98-9d00-48f256240cc9_isrForCrosstalkSources_1125934_51.stderr 2024-03-07 15:08:48 proc_lsst.multi:170 [INFO] [multi] job_id 16896628 proc_lsst.multi INFO: [multi] job_id 16896628 2024-03-07 15:08:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 171 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 171 2024-03-07 15:08:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896628 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896628 parsl.executors.status_handling DEBUG: Launched block 226 on executor multi with job ID 16896628 parsl.executors.status_handling INFO: Allocated block ID 227 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-227 2024-03-07 15:08:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-227 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-227 2024-03-07 15:08:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 171 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 171 2024-03-07 15:08:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:48 proc_lsst.multi:170 [INFO] [multi] job_id 16896629 proc_lsst.multi INFO: [multi] job_id 16896629 2024-03-07 15:08:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 172 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 172 2024-03-07 15:08:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896629 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896629 parsl.executors.status_handling DEBUG: Launched block 227 on executor multi with job ID 16896629 parsl.executors.status_handling INFO: Allocated block ID 228 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-228 2024-03-07 15:08:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-228 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-228 2024-03-07 15:08:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 172 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 172 2024-03-07 15:08:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:49 proc_lsst.multi:170 [INFO] [multi] job_id 16896630 proc_lsst.multi INFO: [multi] job_id 16896630 2024-03-07 15:08:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 173 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 173 2024-03-07 15:08:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896630 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896630 parsl.executors.status_handling DEBUG: Launched block 228 on executor multi with job ID 16896630 parsl.executors.status_handling INFO: Allocated block ID 229 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-229 2024-03-07 15:08:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-229 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-229 2024-03-07 15:08:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 173 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 173 2024-03-07 15:08:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac97522edf8f6796e2e8946c96fef476 with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/abf91d4f-d5ab-47c1-907d-6844d3d80361_isrForCrosstalkSources_1125921_54.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/abf91d4f-d5ab-47c1-907d-6844d3d80361_isrForCrosstalkSources_1125921_54.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5b34b71b51202f3fc5f3d9af9437b79 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/9de91466-495e-47fa-ab67-4a9efae2a1c2_isrForCrosstalkSources_1125891_51.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/9de91466-495e-47fa-ab67-4a9efae2a1c2_isrForCrosstalkSources_1125891_51.stderr 2024-03-07 15:08:49 proc_lsst.multi:170 [INFO] [multi] job_id 16896631 proc_lsst.multi INFO: [multi] job_id 16896631 2024-03-07 15:08:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 174 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 174 2024-03-07 15:08:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896631 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896631 parsl.executors.status_handling DEBUG: Launched block 229 on executor multi with job ID 16896631 parsl.executors.status_handling INFO: Allocated block ID 230 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-230 2024-03-07 15:08:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-230 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-230 2024-03-07 15:08:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 174 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 174 2024-03-07 15:08:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:49 proc_lsst.multi:170 [INFO] [multi] job_id 16896632 proc_lsst.multi INFO: [multi] job_id 16896632 2024-03-07 15:08:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 175 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 175 2024-03-07 15:08:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896632 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896632 parsl.executors.status_handling DEBUG: Launched block 230 on executor multi with job ID 16896632 parsl.executors.status_handling INFO: Allocated block ID 231 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-231 2024-03-07 15:08:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-231 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-231 2024-03-07 15:08:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 175 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 175 2024-03-07 15:08:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 547c8a4216cb93032143747e34edc3a5 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8675e9d7-1dcb-4253-aa22-4f2b944baefa_isrForCrosstalkSources_1125936_38.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8675e9d7-1dcb-4253-aa22-4f2b944baefa_isrForCrosstalkSources_1125936_38.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5aca293e182da6011ea46ac261eeb377 with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/73224468-c9a6-47fc-b9e3-dde6fd88e1a4_isrForCrosstalkSources_1125888_55.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/73224468-c9a6-47fc-b9e3-dde6fd88e1a4_isrForCrosstalkSources_1125888_55.stderr parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1aab2eef0444ee36425da60fbb243677 with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/dc0082f1-178c-442e-8fcb-f5f7f5bec00c_isrForCrosstalkSources_1125911_48.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/dc0082f1-178c-442e-8fcb-f5f7f5bec00c_isrForCrosstalkSources_1125911_48.stderr 2024-03-07 15:08:50 proc_lsst.multi:170 [INFO] [multi] job_id 16896633 proc_lsst.multi INFO: [multi] job_id 16896633 2024-03-07 15:08:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 176 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 176 2024-03-07 15:08:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896633 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896633 parsl.executors.status_handling DEBUG: Launched block 231 on executor multi with job ID 16896633 parsl.executors.status_handling INFO: Allocated block ID 232 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-232 2024-03-07 15:08:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-232 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-232 2024-03-07 15:08:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 176 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 176 2024-03-07 15:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2736feabfdac6ef3c1c19d525c17d8a9 with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/aed14dc1-35e6-4645-afcf-049b7ddfaf54_isrForCrosstalkSources_1125930_61.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/aed14dc1-35e6-4645-afcf-049b7ddfaf54_isrForCrosstalkSources_1125930_61.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df1eabf0bb6e6c5cc8b2fe4bf6a4d561 with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/ed81f9e4-61bd-44aa-bc11-889da5883c0d_isrForCrosstalkSources_1125935_39.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/ed81f9e4-61bd-44aa-bc11-889da5883c0d_isrForCrosstalkSources_1125935_39.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2d53e68e14f8038f39748a966b9b04a with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/28286a84-71c5-482a-a072-4a089290405f_isrForCrosstalkSources_1125911_1.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/28286a84-71c5-482a-a072-4a089290405f_isrForCrosstalkSources_1125911_1.stderr 2024-03-07 15:08:50 proc_lsst.multi:170 [INFO] [multi] job_id 16896634 proc_lsst.multi INFO: [multi] job_id 16896634 2024-03-07 15:08:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 177 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 177 2024-03-07 15:08:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896634 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896634 parsl.executors.status_handling DEBUG: Launched block 232 on executor multi with job ID 16896634 parsl.executors.status_handling INFO: Allocated block ID 233 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-233 2024-03-07 15:08:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-233 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-233 2024-03-07 15:08:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 177 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 177 2024-03-07 15:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=233 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8bd215ac2d9f5fa8957b9290521a778 with result from task 199 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/80f8f990-da38-49ab-9c27-17871a78f425_isrForCrosstalkSources_1125928_39.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/80f8f990-da38-49ab-9c27-17871a78f425_isrForCrosstalkSources_1125928_39.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 775f5d50dca95b9a902ba8137e991bcc with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/0812110b-9530-48b5-9ee5-29eecc353bee_isrForCrosstalkSources_1125896_40.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/0812110b-9530-48b5-9ee5-29eecc353bee_isrForCrosstalkSources_1125896_40.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff7cc7a007bc159f4b1725f1dec34713 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/febbaaf8-e91e-4325-bdb4-25d0f96d4846_isrForCrosstalkSources_1125927_45.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/febbaaf8-e91e-4325-bdb4-25d0f96d4846_isrForCrosstalkSources_1125927_45.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdf9b31bd427ff584c8c60b8eaed683c with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ba0e933c-a86d-4970-8896-9d0cd0b9160b_isrForCrosstalkSources_1125902_55.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ba0e933c-a86d-4970-8896-9d0cd0b9160b_isrForCrosstalkSources_1125902_55.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1f008d9f0c9ad8ee377e3ec83e11533 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/64de8585-672f-4d27-99fb-546a7d737864_isrForCrosstalkSources_1125888_22.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/64de8585-672f-4d27-99fb-546a7d737864_isrForCrosstalkSources_1125888_22.stderr 2024-03-07 15:08:50 proc_lsst.multi:170 [INFO] [multi] job_id 16896635 proc_lsst.multi INFO: [multi] job_id 16896635 2024-03-07 15:08:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 178 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 178 2024-03-07 15:08:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896635 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896635 parsl.executors.status_handling DEBUG: Launched block 233 on executor multi with job ID 16896635 parsl.executors.status_handling INFO: Allocated block ID 234 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-234 2024-03-07 15:08:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-234 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-234 2024-03-07 15:08:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 178 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 178 2024-03-07 15:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=234 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d13f86bd4ef014a7d71b48adca17c269 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/9075aa40-0841-4f95-a580-0816351cc4ed_isrForCrosstalkSources_1125926_20.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/9075aa40-0841-4f95-a580-0816351cc4ed_isrForCrosstalkSources_1125926_20.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08b21abe007f97ca1550dd30cba98023 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/607cb5fe-0683-4024-94c6-9ce00bbae060_isrForCrosstalkSources_1125912_5.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/607cb5fe-0683-4024-94c6-9ce00bbae060_isrForCrosstalkSources_1125912_5.stderr 2024-03-07 15:08:51 proc_lsst.multi:170 [INFO] [multi] job_id 16896636 proc_lsst.multi INFO: [multi] job_id 16896636 2024-03-07 15:08:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 179 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 179 2024-03-07 15:08:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896636 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896636 parsl.executors.status_handling DEBUG: Launched block 234 on executor multi with job ID 16896636 parsl.executors.status_handling INFO: Allocated block ID 235 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-235 2024-03-07 15:08:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-235 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-235 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 179 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 179 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=235 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:51 proc_lsst.multi:170 [INFO] [multi] job_id 16896637 proc_lsst.multi INFO: [multi] job_id 16896637 2024-03-07 15:08:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 180 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 180 2024-03-07 15:08:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896637 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896637 parsl.executors.status_handling DEBUG: Launched block 235 on executor multi with job ID 16896637 parsl.executors.status_handling INFO: Allocated block ID 236 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-236 2024-03-07 15:08:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-236 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-236 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 180 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 180 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=236 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f70d5ab168b600d7c99c99f034b769d6 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/bb9272c3-83fc-4874-8efe-6460b5813de1_isrForCrosstalkSources_1125935_62.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/bb9272c3-83fc-4874-8efe-6460b5813de1_isrForCrosstalkSources_1125935_62.stderr 2024-03-07 15:08:51 proc_lsst.multi:170 [INFO] [multi] job_id 16896638 proc_lsst.multi INFO: [multi] job_id 16896638 2024-03-07 15:08:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 181 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 181 2024-03-07 15:08:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896638 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896638 parsl.executors.status_handling DEBUG: Launched block 236 on executor multi with job ID 16896638 parsl.executors.status_handling INFO: Allocated block ID 237 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-237 2024-03-07 15:08:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-237 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-237 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 181 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 181 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=237 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:51 proc_lsst.multi:170 [INFO] [multi] job_id 16896639 proc_lsst.multi INFO: [multi] job_id 16896639 2024-03-07 15:08:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 182 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 182 2024-03-07 15:08:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896639 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896639 parsl.executors.status_handling DEBUG: Launched block 237 on executor multi with job ID 16896639 parsl.executors.status_handling INFO: Allocated block ID 238 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-238 2024-03-07 15:08:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-238 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-238 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 182 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 182 2024-03-07 15:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=238 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:52 proc_lsst.multi:170 [INFO] [multi] job_id 16896640 proc_lsst.multi INFO: [multi] job_id 16896640 2024-03-07 15:08:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 183 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 183 2024-03-07 15:08:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896640 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896640 parsl.executors.status_handling DEBUG: Launched block 238 on executor multi with job ID 16896640 parsl.executors.status_handling INFO: Allocated block ID 239 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-239 2024-03-07 15:08:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-239 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-239 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 183 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 183 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=239 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d22a27dd6cb050f632359179dce4af1 with result from task 217 2024-03-07 15:08:52 proc_lsst.multi:170 [INFO] [multi] job_id 16896641 proc_lsst.multi INFO: [multi] job_id 16896641 2024-03-07 15:08:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 184 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 184 2024-03-07 15:08:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896641 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896641 parsl.executors.status_handling DEBUG: Launched block 239 on executor multi with job ID 16896641 parsl.executors.status_handling INFO: Allocated block ID 240 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-240 2024-03-07 15:08:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-240 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-240 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 184 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 184 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=240 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/dae06d16-a9c0-43e3-b974-f7d725ef0275_isrForCrosstalkSources_1125926_15.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/dae06d16-a9c0-43e3-b974-f7d725ef0275_isrForCrosstalkSources_1125926_15.stderr parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:52 proc_lsst.multi:170 [INFO] [multi] job_id 16896642 proc_lsst.multi INFO: [multi] job_id 16896642 2024-03-07 15:08:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 185 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 185 2024-03-07 15:08:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896642 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896642 parsl.executors.status_handling DEBUG: Launched block 240 on executor multi with job ID 16896642 parsl.executors.status_handling INFO: Allocated block ID 241 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-241 2024-03-07 15:08:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-241 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-241 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 185 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 185 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=241 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a997fede114fcb666bce3f46f950efe5 with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/62c825b9-92d0-4620-8c67-6ad44d3730b3_isrForCrosstalkSources_1125938_11.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/62c825b9-92d0-4620-8c67-6ad44d3730b3_isrForCrosstalkSources_1125938_11.stderr parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:52 proc_lsst.multi:170 [INFO] [multi] job_id 16896643 proc_lsst.multi INFO: [multi] job_id 16896643 2024-03-07 15:08:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 186 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 186 2024-03-07 15:08:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896643 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896643 parsl.executors.status_handling DEBUG: Launched block 241 on executor multi with job ID 16896643 parsl.executors.status_handling INFO: Allocated block ID 242 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-242 2024-03-07 15:08:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-242 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-242 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 186 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 186 2024-03-07 15:08:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=242 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 970df926b9da5f39f03e9736ae637b4e with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/8592b626-8983-4967-8fe5-453baedb3676_isrForCrosstalkSources_1125917_25.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/8592b626-8983-4967-8fe5-453baedb3676_isrForCrosstalkSources_1125917_25.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70bdfe98ea1fc1d2f860a056be74c6fe with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/54556909-eaab-47c3-a3a2-0ece8cf1e3a3_isrForCrosstalkSources_1125889_19.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/54556909-eaab-47c3-a3a2-0ece8cf1e3a3_isrForCrosstalkSources_1125889_19.stderr 2024-03-07 15:08:53 proc_lsst.multi:170 [INFO] [multi] job_id 16896644 proc_lsst.multi INFO: [multi] job_id 16896644 2024-03-07 15:08:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 187 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 187 2024-03-07 15:08:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896644 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896644 parsl.executors.status_handling DEBUG: Launched block 242 on executor multi with job ID 16896644 parsl.executors.status_handling INFO: Allocated block ID 243 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-243 2024-03-07 15:08:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-243 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-243 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 187 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 187 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=243 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecb3f97659899f2832d35bba5f612322 with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/a894a6db-3988-4802-a39a-0938ef572199_isrForCrosstalkSources_1125914_33.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/a894a6db-3988-4802-a39a-0938ef572199_isrForCrosstalkSources_1125914_33.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5398a87a31f94780527da17bbc7a392 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/3a1ca9d3-2075-47ba-a1d1-e460b38a03f1_isrForCrosstalkSources_1125917_18.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/3a1ca9d3-2075-47ba-a1d1-e460b38a03f1_isrForCrosstalkSources_1125917_18.stderr 2024-03-07 15:08:53 proc_lsst.multi:170 [INFO] [multi] job_id 16896645 proc_lsst.multi INFO: [multi] job_id 16896645 2024-03-07 15:08:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 188 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 188 2024-03-07 15:08:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896645 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896645 parsl.executors.status_handling DEBUG: Launched block 243 on executor multi with job ID 16896645 parsl.executors.status_handling INFO: Allocated block ID 244 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-244 2024-03-07 15:08:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-244 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-244 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 188 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 188 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=244 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:53 proc_lsst.multi:170 [INFO] [multi] job_id 16896646 proc_lsst.multi INFO: [multi] job_id 16896646 2024-03-07 15:08:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 189 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 189 2024-03-07 15:08:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896646 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896646 parsl.executors.status_handling DEBUG: Launched block 244 on executor multi with job ID 16896646 parsl.executors.status_handling INFO: Allocated block ID 245 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-245 2024-03-07 15:08:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-245 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-245 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 189 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 189 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=245 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:53 proc_lsst.multi:170 [INFO] [multi] job_id 16896647 proc_lsst.multi INFO: [multi] job_id 16896647 2024-03-07 15:08:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 190 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 190 2024-03-07 15:08:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896647 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896647 parsl.executors.status_handling DEBUG: Launched block 245 on executor multi with job ID 16896647 parsl.executors.status_handling INFO: Allocated block ID 246 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-246 2024-03-07 15:08:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-246 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-246 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 190 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 190 2024-03-07 15:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=246 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06089d4292f09b18507ed354f513529b with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/714b4f08-0973-4c61-acde-fcb21312debd_isrForCrosstalkSources_1125933_14.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/714b4f08-0973-4c61-acde-fcb21312debd_isrForCrosstalkSources_1125933_14.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed9180629fbd6966a4dd5a651c6ae0d1 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0632ac0f-2158-4411-9d7f-133862ff2146_isrForCrosstalkSources_1125904_57.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0632ac0f-2158-4411-9d7f-133862ff2146_isrForCrosstalkSources_1125904_57.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd37b23ec9728e9b3ea6f30c51cd2053 with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/6795e208-7e09-41b6-acaa-aea6b8a6f407_isrForCrosstalkSources_1125896_38.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/6795e208-7e09-41b6-acaa-aea6b8a6f407_isrForCrosstalkSources_1125896_38.stderr 2024-03-07 15:08:54 proc_lsst.multi:170 [INFO] [multi] job_id 16896648 proc_lsst.multi INFO: [multi] job_id 16896648 2024-03-07 15:08:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 191 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 191 2024-03-07 15:08:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896648 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896648 parsl.executors.status_handling DEBUG: Launched block 246 on executor multi with job ID 16896648 parsl.executors.status_handling INFO: Allocated block ID 247 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-247 2024-03-07 15:08:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-247 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-247 2024-03-07 15:08:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 191 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 191 2024-03-07 15:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=247 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b2ccc6d39b4748c8c2b6ec280b86343 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/289670df-1c85-4ee0-aed2-7d506c292acb_isrForCrosstalkSources_1125937_60.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/289670df-1c85-4ee0-aed2-7d506c292acb_isrForCrosstalkSources_1125937_60.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d87daa789628339f4b9f6575fe5634eb with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d7033dc8-3829-4179-b9f5-338205e635e0_isrForCrosstalkSources_1125934_5.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d7033dc8-3829-4179-b9f5-338205e635e0_isrForCrosstalkSources_1125934_5.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b24e9ff304812444a10e558e8893ce4e with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/1beff0ee-ec59-48ce-8ab2-9e25ab3c3b65_isrForCrosstalkSources_1125920_39.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/1beff0ee-ec59-48ce-8ab2-9e25ab3c3b65_isrForCrosstalkSources_1125920_39.stderr 2024-03-07 15:08:54 proc_lsst.multi:170 [INFO] [multi] job_id 16896649 proc_lsst.multi INFO: [multi] job_id 16896649 2024-03-07 15:08:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 192 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 192 2024-03-07 15:08:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896649 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896649 parsl.executors.status_handling DEBUG: Launched block 247 on executor multi with job ID 16896649 parsl.executors.status_handling INFO: Allocated block ID 248 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-248 2024-03-07 15:08:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-248 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-248 2024-03-07 15:08:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 192 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 192 2024-03-07 15:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=248 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e22536481f6abc9c04332fa6e40538e4 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/1d14e3ea-8303-422e-a3a0-e19641cbb05a_isrForCrosstalkSources_1125919_5.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/1d14e3ea-8303-422e-a3a0-e19641cbb05a_isrForCrosstalkSources_1125919_5.stderr 2024-03-07 15:08:54 proc_lsst.multi:170 [INFO] [multi] job_id 16896650 proc_lsst.multi INFO: [multi] job_id 16896650 2024-03-07 15:08:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 193 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 193 2024-03-07 15:08:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896650 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896650 parsl.executors.status_handling DEBUG: Launched block 248 on executor multi with job ID 16896650 parsl.executors.status_handling INFO: Allocated block ID 249 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-249 2024-03-07 15:08:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-249 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-249 2024-03-07 15:08:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 193 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 193 2024-03-07 15:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=249 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:55 proc_lsst.multi:170 [INFO] [multi] job_id 16896651 proc_lsst.multi INFO: [multi] job_id 16896651 2024-03-07 15:08:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 194 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 194 2024-03-07 15:08:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896651 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896651 parsl.executors.status_handling DEBUG: Launched block 249 on executor multi with job ID 16896651 parsl.executors.status_handling INFO: Allocated block ID 250 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-250 2024-03-07 15:08:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-250 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-250 2024-03-07 15:08:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 194 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 194 2024-03-07 15:08:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=250 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:55 proc_lsst.multi:170 [INFO] [multi] job_id 16896652 proc_lsst.multi INFO: [multi] job_id 16896652 2024-03-07 15:08:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 195 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 195 2024-03-07 15:08:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896652 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896652 parsl.executors.status_handling DEBUG: Launched block 250 on executor multi with job ID 16896652 parsl.executors.status_handling INFO: Allocated block ID 251 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-251 2024-03-07 15:08:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-251 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-251 2024-03-07 15:08:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 195 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 195 2024-03-07 15:08:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=251 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 069fa67b44531a5c9788a32f018748ba with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/33495f80-52c0-4559-8cb0-d5b44d2d5703_isrForCrosstalkSources_1125922_25.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/33495f80-52c0-4559-8cb0-d5b44d2d5703_isrForCrosstalkSources_1125922_25.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf313e8c6d4d84a1f72c56b4c3f2fffc with result from task 214 2024-03-07 15:08:55 proc_lsst.multi:170 [INFO] [multi] job_id 16896653 proc_lsst.multi INFO: [multi] job_id 16896653 2024-03-07 15:08:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 196 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 196 2024-03-07 15:08:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896653 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896653 parsl.executors.status_handling DEBUG: Launched block 251 on executor multi with job ID 16896653 parsl.executors.status_handling INFO: Allocated block ID 252 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-252 2024-03-07 15:08:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-252 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-252 2024-03-07 15:08:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 196 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 196 2024-03-07 15:08:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=252 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/8614975c-e20b-4bbc-b5d9-6c2f4e4d63ba_isrForCrosstalkSources_1125920_8.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/8614975c-e20b-4bbc-b5d9-6c2f4e4d63ba_isrForCrosstalkSources_1125920_8.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 041c2686b0a58cbefc8f705ffe02e39b with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/bc8db282-96bf-434c-bc81-75d19b7cc731_isrForCrosstalkSources_1125902_46.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/bc8db282-96bf-434c-bc81-75d19b7cc731_isrForCrosstalkSources_1125902_46.stderr 2024-03-07 15:08:56 proc_lsst.multi:170 [INFO] [multi] job_id 16896654 proc_lsst.multi INFO: [multi] job_id 16896654 2024-03-07 15:08:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 197 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 197 2024-03-07 15:08:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896654 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896654 parsl.executors.status_handling DEBUG: Launched block 252 on executor multi with job ID 16896654 parsl.executors.status_handling INFO: Allocated block ID 253 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-253 2024-03-07 15:08:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-253 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-253 2024-03-07 15:08:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 197 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 197 2024-03-07 15:08:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=253 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:56 proc_lsst.multi:170 [INFO] [multi] job_id 16896655 proc_lsst.multi INFO: [multi] job_id 16896655 2024-03-07 15:08:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 198 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 198 2024-03-07 15:08:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896655 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896655 parsl.executors.status_handling DEBUG: Launched block 253 on executor multi with job ID 16896655 parsl.executors.status_handling INFO: Allocated block ID 254 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-254 2024-03-07 15:08:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-254 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-254 2024-03-07 15:08:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 198 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 198 2024-03-07 15:08:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=254 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:57 proc_lsst.multi:170 [INFO] [multi] job_id 16896656 proc_lsst.multi INFO: [multi] job_id 16896656 2024-03-07 15:08:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 199 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 199 2024-03-07 15:08:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896656 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896656 parsl.executors.status_handling DEBUG: Launched block 254 on executor multi with job ID 16896656 parsl.executors.status_handling INFO: Allocated block ID 255 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-255 2024-03-07 15:08:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-255 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-255 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 199 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 199 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=255 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:57 proc_lsst.multi:170 [INFO] [multi] job_id 16896657 proc_lsst.multi INFO: [multi] job_id 16896657 2024-03-07 15:08:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 200 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 200 2024-03-07 15:08:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896657 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896657 parsl.executors.status_handling DEBUG: Launched block 255 on executor multi with job ID 16896657 parsl.executors.status_handling INFO: Allocated block ID 256 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-256 2024-03-07 15:08:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-256 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-256 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 200 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 200 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=256 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c4969c8673f4a85a186e080ca3e49b5 with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/d75e5107-5992-419e-a5c1-8fe48293f49c_isrForCrosstalkSources_1125931_47.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/d75e5107-5992-419e-a5c1-8fe48293f49c_isrForCrosstalkSources_1125931_47.stderr 2024-03-07 15:08:57 proc_lsst.multi:170 [INFO] [multi] job_id 16896658 proc_lsst.multi INFO: [multi] job_id 16896658 2024-03-07 15:08:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 201 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 201 2024-03-07 15:08:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896658 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896658 parsl.executors.status_handling DEBUG: Launched block 256 on executor multi with job ID 16896658 parsl.executors.status_handling INFO: Allocated block ID 257 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-257 2024-03-07 15:08:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-257 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-257 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 201 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 201 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=257 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:57 proc_lsst.multi:170 [INFO] [multi] job_id 16896659 proc_lsst.multi INFO: [multi] job_id 16896659 2024-03-07 15:08:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 202 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 202 2024-03-07 15:08:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896659 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896659 parsl.executors.status_handling DEBUG: Launched block 257 on executor multi with job ID 16896659 parsl.executors.status_handling INFO: Allocated block ID 258 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-258 2024-03-07 15:08:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-258 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-258 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 202 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 202 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=258 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:57 proc_lsst.multi:170 [INFO] [multi] job_id 16896660 proc_lsst.multi INFO: [multi] job_id 16896660 2024-03-07 15:08:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 203 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 203 2024-03-07 15:08:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896660 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896660 parsl.executors.status_handling DEBUG: Launched block 258 on executor multi with job ID 16896660 parsl.executors.status_handling INFO: Allocated block ID 259 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-259 2024-03-07 15:08:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-259 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-259 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 203 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 203 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=259 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:57 proc_lsst.multi:170 [INFO] [multi] job_id 16896661 proc_lsst.multi INFO: [multi] job_id 16896661 2024-03-07 15:08:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 204 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 204 2024-03-07 15:08:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896661 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896661 parsl.executors.status_handling DEBUG: Launched block 259 on executor multi with job ID 16896661 parsl.executors.status_handling INFO: Allocated block ID 260 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-260 2024-03-07 15:08:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-260 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-260 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 204 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 204 2024-03-07 15:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=260 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:58 proc_lsst.multi:170 [INFO] [multi] job_id 16896662 proc_lsst.multi INFO: [multi] job_id 16896662 2024-03-07 15:08:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 205 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 205 2024-03-07 15:08:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896662 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896662 parsl.executors.status_handling DEBUG: Launched block 260 on executor multi with job ID 16896662 parsl.executors.status_handling INFO: Allocated block ID 261 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-261 2024-03-07 15:08:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-261 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-261 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 205 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 205 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=261 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:58 proc_lsst.multi:170 [INFO] [multi] job_id 16896663 proc_lsst.multi INFO: [multi] job_id 16896663 2024-03-07 15:08:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 206 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 206 2024-03-07 15:08:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896663 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896663 parsl.executors.status_handling DEBUG: Launched block 261 on executor multi with job ID 16896663 parsl.executors.status_handling INFO: Allocated block ID 262 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-262 2024-03-07 15:08:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-262 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-262 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 206 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 206 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=262 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:58 proc_lsst.multi:170 [INFO] [multi] job_id 16896664 proc_lsst.multi INFO: [multi] job_id 16896664 2024-03-07 15:08:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 207 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 207 2024-03-07 15:08:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896664 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896664 parsl.executors.status_handling DEBUG: Launched block 262 on executor multi with job ID 16896664 parsl.executors.status_handling INFO: Allocated block ID 263 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-263 2024-03-07 15:08:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-263 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-263 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 207 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 207 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=263 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:58 proc_lsst.multi:170 [INFO] [multi] job_id 16896665 proc_lsst.multi INFO: [multi] job_id 16896665 2024-03-07 15:08:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 208 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 208 2024-03-07 15:08:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896665 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896665 parsl.executors.status_handling DEBUG: Launched block 263 on executor multi with job ID 16896665 parsl.executors.status_handling INFO: Allocated block ID 264 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-264 2024-03-07 15:08:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-264 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-264 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 208 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 208 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=264 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:58 proc_lsst.multi:170 [INFO] [multi] job_id 16896666 proc_lsst.multi INFO: [multi] job_id 16896666 2024-03-07 15:08:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 209 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 209 2024-03-07 15:08:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896666 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896666 parsl.executors.status_handling DEBUG: Launched block 264 on executor multi with job ID 16896666 parsl.executors.status_handling INFO: Allocated block ID 265 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-265 2024-03-07 15:08:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-265 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-265 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 209 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 209 2024-03-07 15:08:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=265 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:59 proc_lsst.multi:170 [INFO] [multi] job_id 16896667 proc_lsst.multi INFO: [multi] job_id 16896667 2024-03-07 15:08:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 210 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 210 2024-03-07 15:08:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896667 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896667 parsl.executors.status_handling DEBUG: Launched block 265 on executor multi with job ID 16896667 parsl.executors.status_handling INFO: Allocated block ID 266 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-266 2024-03-07 15:08:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-266 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-266 2024-03-07 15:08:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 210 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 210 2024-03-07 15:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=266 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:59 proc_lsst.multi:170 [INFO] [multi] job_id 16896668 proc_lsst.multi INFO: [multi] job_id 16896668 2024-03-07 15:08:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 211 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 211 2024-03-07 15:08:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896668 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896668 parsl.executors.status_handling DEBUG: Launched block 266 on executor multi with job ID 16896668 parsl.executors.status_handling INFO: Allocated block ID 267 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-267 2024-03-07 15:08:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-267 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-267 2024-03-07 15:08:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 211 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 211 2024-03-07 15:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=267 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:08:59 proc_lsst.multi:170 [INFO] [multi] job_id 16896669 proc_lsst.multi INFO: [multi] job_id 16896669 2024-03-07 15:08:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 212 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 212 2024-03-07 15:08:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896669 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896669 parsl.executors.status_handling DEBUG: Launched block 267 on executor multi with job ID 16896669 parsl.executors.status_handling INFO: Allocated block ID 268 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-268 2024-03-07 15:08:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-268 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-268 2024-03-07 15:08:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:08:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:08:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:08:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:08:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 212 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 212 2024-03-07 15:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:08:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=268 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c642d84df6aade5a5c4f8d7cb7683052 with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/72ecf0af-23e9-4dfb-8439-ae051d55721e_isrForCrosstalkSources_1125906_33.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/72ecf0af-23e9-4dfb-8439-ae051d55721e_isrForCrosstalkSources_1125906_33.stderr 2024-03-07 15:09:00 proc_lsst.multi:170 [INFO] [multi] job_id 16896670 proc_lsst.multi INFO: [multi] job_id 16896670 2024-03-07 15:09:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 213 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 213 2024-03-07 15:09:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896670 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896670 parsl.executors.status_handling DEBUG: Launched block 268 on executor multi with job ID 16896670 parsl.executors.status_handling INFO: Allocated block ID 269 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-269 2024-03-07 15:09:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-269 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-269 2024-03-07 15:09:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 213 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 213 2024-03-07 15:09:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=269 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ad2feb878c9fa6fae30db02569cfed6 with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/a8a97aa1-f76b-43d6-91d4-83601e208319_isrForCrosstalkSources_1125906_19.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/a8a97aa1-f76b-43d6-91d4-83601e208319_isrForCrosstalkSources_1125906_19.stderr 2024-03-07 15:09:00 proc_lsst.multi:170 [INFO] [multi] job_id 16896671 proc_lsst.multi INFO: [multi] job_id 16896671 2024-03-07 15:09:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 214 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 214 2024-03-07 15:09:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896671 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896671 parsl.executors.status_handling DEBUG: Launched block 269 on executor multi with job ID 16896671 parsl.executors.status_handling INFO: Allocated block ID 270 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-270 2024-03-07 15:09:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-270 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-270 2024-03-07 15:09:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 214 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 214 2024-03-07 15:09:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=270 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef857a8b370b645c7db6792abf9cbd15 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/1f635357-0ce4-40f5-996f-a2324e877a50_isrForCrosstalkSources_1125938_40.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/1f635357-0ce4-40f5-996f-a2324e877a50_isrForCrosstalkSources_1125938_40.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc7899e542f7034a526b4ecf197acb55 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/bbb5b755-f86b-4b56-9212-e74c026de731_isrForCrosstalkSources_1125917_56.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/bbb5b755-f86b-4b56-9212-e74c026de731_isrForCrosstalkSources_1125917_56.stderr 2024-03-07 15:09:00 proc_lsst.multi:170 [INFO] [multi] job_id 16896672 proc_lsst.multi INFO: [multi] job_id 16896672 2024-03-07 15:09:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 215 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 215 2024-03-07 15:09:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896672 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896672 parsl.executors.status_handling DEBUG: Launched block 270 on executor multi with job ID 16896672 parsl.executors.status_handling INFO: Allocated block ID 271 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-271 2024-03-07 15:09:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-271 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-271 2024-03-07 15:09:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 215 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 215 2024-03-07 15:09:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=271 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9cf4f2b25d087268290417f19e5db3d with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/8a4df76e-5fae-496a-adf3-66c29d1f96a1_isrForCrosstalkSources_1125888_6.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/8a4df76e-5fae-496a-adf3-66c29d1f96a1_isrForCrosstalkSources_1125888_6.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a4eef9970a44d6a98c104722d8e4de0 with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3b3c7554-bfef-4512-98ea-729c71419982_isrForCrosstalkSources_1125920_10.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3b3c7554-bfef-4512-98ea-729c71419982_isrForCrosstalkSources_1125920_10.stderr 2024-03-07 15:09:01 proc_lsst.multi:170 [INFO] [multi] job_id 16896673 proc_lsst.multi INFO: [multi] job_id 16896673 2024-03-07 15:09:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 216 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 216 2024-03-07 15:09:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896673 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896673 parsl.executors.status_handling DEBUG: Launched block 271 on executor multi with job ID 16896673 parsl.executors.status_handling INFO: Allocated block ID 272 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-272 2024-03-07 15:09:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-272 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-272 2024-03-07 15:09:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 216 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 216 2024-03-07 15:09:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=272 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31b5d7602de4ec5827760f9527e51b8c with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e29dc6fa-6f9d-44b4-8cdc-a7c71f3cfc35_isrForCrosstalkSources_1125895_10.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e29dc6fa-6f9d-44b4-8cdc-a7c71f3cfc35_isrForCrosstalkSources_1125895_10.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdd7a945822642b45e4a12a7f62f0370 with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0b06d4dd-bad0-4db8-a0a8-aaef9eb0f455_isrForCrosstalkSources_1125914_36.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0b06d4dd-bad0-4db8-a0a8-aaef9eb0f455_isrForCrosstalkSources_1125914_36.stderr 2024-03-07 15:09:01 proc_lsst.multi:170 [INFO] [multi] job_id 16896674 proc_lsst.multi INFO: [multi] job_id 16896674 2024-03-07 15:09:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 217 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 217 2024-03-07 15:09:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896674 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896674 parsl.executors.status_handling DEBUG: Launched block 272 on executor multi with job ID 16896674 parsl.executors.status_handling INFO: Allocated block ID 273 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-273 2024-03-07 15:09:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-273 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-273 2024-03-07 15:09:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 217 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 217 2024-03-07 15:09:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=273 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e631f4d037b07b6c5d3545b601ecd40 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/62ea93e6-9cd2-43e7-aa45-96440f82c2bc_isrForCrosstalkSources_1125890_35.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/62ea93e6-9cd2-43e7-aa45-96440f82c2bc_isrForCrosstalkSources_1125890_35.stderr parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58b36801d08328c4f7b5bc3f6552db8a with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/22c2dda9-1ebd-41aa-a3cd-21715b9134df_isrForCrosstalkSources_1125915_25.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/22c2dda9-1ebd-41aa-a3cd-21715b9134df_isrForCrosstalkSources_1125915_25.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92b1bd1b37954fbe29ef82af12c9dc88 with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/81f53f79-b7e0-44cf-8b98-659fc2485b9e_isrForCrosstalkSources_1125903_5.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/81f53f79-b7e0-44cf-8b98-659fc2485b9e_isrForCrosstalkSources_1125903_5.stderr parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d52d60a34da3d7b3b7391aa98bf139d with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/25a3a2fe-d705-4bf9-aa68-e861b114a253_isrForCrosstalkSources_1125919_59.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/25a3a2fe-d705-4bf9-aa68-e861b114a253_isrForCrosstalkSources_1125919_59.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a67f76efce23700a00b1d2cd98382fe with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/f60ff89d-1205-4d89-9cae-8966bee255a8_isrForCrosstalkSources_1125931_43.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/f60ff89d-1205-4d89-9cae-8966bee255a8_isrForCrosstalkSources_1125931_43.stderr parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a825bd268b5e98bcd6860000a0446ea5 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/c3fd174b-ba59-4845-bc3a-a4403d553fca_isrForCrosstalkSources_1125895_59.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/c3fd174b-ba59-4845-bc3a-a4403d553fca_isrForCrosstalkSources_1125895_59.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5105d77466551aa93122cf4a6e91052d with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/63305e4a-1bb0-4f25-86ce-7d38ea7d08d8_isrForCrosstalkSources_1125893_33.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/63305e4a-1bb0-4f25-86ce-7d38ea7d08d8_isrForCrosstalkSources_1125893_33.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b06d67dc72de3da77765550e4b1a4a1c with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/0e64442e-5319-4a1d-8e2d-a9ab2971b13a_isrForCrosstalkSources_1125917_60.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/0e64442e-5319-4a1d-8e2d-a9ab2971b13a_isrForCrosstalkSources_1125917_60.stderr 2024-03-07 15:09:03 proc_lsst.multi:170 [INFO] [multi] job_id 16896675 proc_lsst.multi INFO: [multi] job_id 16896675 2024-03-07 15:09:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 218 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 218 2024-03-07 15:09:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896675 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896675 parsl.executors.status_handling DEBUG: Launched block 273 on executor multi with job ID 16896675 parsl.executors.status_handling INFO: Allocated block ID 274 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-274 2024-03-07 15:09:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-274 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-274 2024-03-07 15:09:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 218 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 218 2024-03-07 15:09:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=274 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:03 proc_lsst.multi:170 [INFO] [multi] job_id 16896676 proc_lsst.multi INFO: [multi] job_id 16896676 2024-03-07 15:09:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 219 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 219 2024-03-07 15:09:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896676 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896676 parsl.executors.status_handling DEBUG: Launched block 274 on executor multi with job ID 16896676 parsl.executors.status_handling INFO: Allocated block ID 275 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-275 2024-03-07 15:09:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-275 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-275 2024-03-07 15:09:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 219 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 219 2024-03-07 15:09:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=275 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e47a6f140a8c1738246771a635016f9c with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/29d49340-ac61-480b-848f-76f9cd85e1c4_isrForCrosstalkSources_1125937_21.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/29d49340-ac61-480b-848f-76f9cd85e1c4_isrForCrosstalkSources_1125937_21.stderr 2024-03-07 15:09:04 proc_lsst.multi:170 [INFO] [multi] job_id 16896678 proc_lsst.multi INFO: [multi] job_id 16896678 2024-03-07 15:09:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 220 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 220 2024-03-07 15:09:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896678 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896678 parsl.executors.status_handling DEBUG: Launched block 275 on executor multi with job ID 16896678 parsl.executors.status_handling INFO: Allocated block ID 276 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-276 2024-03-07 15:09:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-276 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-276 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 220 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 220 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=276 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:04 proc_lsst.multi:170 [INFO] [multi] job_id 16896679 proc_lsst.multi INFO: [multi] job_id 16896679 2024-03-07 15:09:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 221 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 221 2024-03-07 15:09:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896679 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896679 parsl.executors.status_handling DEBUG: Launched block 276 on executor multi with job ID 16896679 parsl.executors.status_handling INFO: Allocated block ID 277 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-277 2024-03-07 15:09:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-277 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-277 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 221 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 221 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=277 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8df181076b328706ae3ac181baad6b2c with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/ade256fd-ee0f-4379-9f84-59ee4702c62f_isrForCrosstalkSources_1125937_56.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/ade256fd-ee0f-4379-9f84-59ee4702c62f_isrForCrosstalkSources_1125937_56.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3959dbdbec1a3aa78df00b2af279f322 with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/6f71ab4a-897b-443f-aa15-871a55cba00e_isrForCrosstalkSources_1125907_23.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/6f71ab4a-897b-443f-aa15-871a55cba00e_isrForCrosstalkSources_1125907_23.stderr 2024-03-07 15:09:04 proc_lsst.multi:170 [INFO] [multi] job_id 16896680 proc_lsst.multi INFO: [multi] job_id 16896680 2024-03-07 15:09:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 222 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 222 2024-03-07 15:09:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896680 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896680 parsl.executors.status_handling DEBUG: Launched block 277 on executor multi with job ID 16896680 parsl.executors.status_handling INFO: Allocated block ID 278 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-278 2024-03-07 15:09:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-278 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-278 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 222 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 222 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=278 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b9b73a21a7c5e4d506553671469a4cd with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ff308b62-8263-42a3-9c17-eaf1680f4cd1_isrForCrosstalkSources_1125902_16.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ff308b62-8263-42a3-9c17-eaf1680f4cd1_isrForCrosstalkSources_1125902_16.stderr 2024-03-07 15:09:04 proc_lsst.multi:170 [INFO] [multi] job_id 16896681 proc_lsst.multi INFO: [multi] job_id 16896681 2024-03-07 15:09:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 223 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 223 2024-03-07 15:09:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896681 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896681 parsl.executors.status_handling DEBUG: Launched block 278 on executor multi with job ID 16896681 parsl.executors.status_handling INFO: Allocated block ID 279 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-279 2024-03-07 15:09:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-279 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-279 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 223 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 223 2024-03-07 15:09:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=279 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:05 proc_lsst.multi:170 [INFO] [multi] job_id 16896682 proc_lsst.multi INFO: [multi] job_id 16896682 2024-03-07 15:09:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 224 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 224 2024-03-07 15:09:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896682 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896682 parsl.executors.status_handling DEBUG: Launched block 279 on executor multi with job ID 16896682 parsl.executors.status_handling INFO: Allocated block ID 280 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-280 2024-03-07 15:09:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-280 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-280 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 224 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 224 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=280 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:05 proc_lsst.multi:170 [INFO] [multi] job_id 16896683 proc_lsst.multi INFO: [multi] job_id 16896683 2024-03-07 15:09:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 225 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 225 2024-03-07 15:09:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896683 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896683 parsl.executors.status_handling DEBUG: Launched block 280 on executor multi with job ID 16896683 parsl.executors.status_handling INFO: Allocated block ID 281 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-281 2024-03-07 15:09:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-281 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-281 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 225 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 225 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=281 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:05 proc_lsst.multi:170 [INFO] [multi] job_id 16896684 proc_lsst.multi INFO: [multi] job_id 16896684 2024-03-07 15:09:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 226 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 226 2024-03-07 15:09:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896684 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896684 parsl.executors.status_handling DEBUG: Launched block 281 on executor multi with job ID 16896684 parsl.executors.status_handling INFO: Allocated block ID 282 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-282 2024-03-07 15:09:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-282 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-282 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 226 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 226 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=282 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:05 proc_lsst.multi:170 [INFO] [multi] job_id 16896685 proc_lsst.multi INFO: [multi] job_id 16896685 2024-03-07 15:09:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 227 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 227 2024-03-07 15:09:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896685 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896685 parsl.executors.status_handling DEBUG: Launched block 282 on executor multi with job ID 16896685 parsl.executors.status_handling INFO: Allocated block ID 283 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-283 2024-03-07 15:09:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-283 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-283 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 227 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 227 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=283 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:05 proc_lsst.multi:170 [INFO] [multi] job_id 16896686 proc_lsst.multi INFO: [multi] job_id 16896686 2024-03-07 15:09:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 228 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 228 2024-03-07 15:09:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896686 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896686 parsl.executors.status_handling DEBUG: Launched block 283 on executor multi with job ID 16896686 parsl.executors.status_handling INFO: Allocated block ID 284 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-284 2024-03-07 15:09:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-284 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-284 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 228 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 228 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=284 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:05 proc_lsst.multi:170 [INFO] [multi] job_id 16896687 proc_lsst.multi INFO: [multi] job_id 16896687 2024-03-07 15:09:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 229 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 229 2024-03-07 15:09:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896687 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896687 parsl.executors.status_handling DEBUG: Launched block 284 on executor multi with job ID 16896687 parsl.executors.status_handling INFO: Allocated block ID 285 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-285 2024-03-07 15:09:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-285 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-285 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 229 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 229 2024-03-07 15:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=285 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:06 proc_lsst.multi:170 [INFO] [multi] job_id 16896688 proc_lsst.multi INFO: [multi] job_id 16896688 2024-03-07 15:09:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 230 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 230 2024-03-07 15:09:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896688 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896688 parsl.executors.status_handling DEBUG: Launched block 285 on executor multi with job ID 16896688 parsl.executors.status_handling INFO: Allocated block ID 286 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-286 2024-03-07 15:09:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-286 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-286 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 230 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 230 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=286 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80bbfa09475caa4ca80bbe35f0c1c042 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/a0cbc1d6-3f9d-456b-a0e9-9040738f36ab_isrForCrosstalkSources_1125890_37.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/a0cbc1d6-3f9d-456b-a0e9-9040738f36ab_isrForCrosstalkSources_1125890_37.stderr 2024-03-07 15:09:06 proc_lsst.multi:170 [INFO] [multi] job_id 16896689 proc_lsst.multi INFO: [multi] job_id 16896689 2024-03-07 15:09:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 231 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 231 2024-03-07 15:09:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896689 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896689 parsl.executors.status_handling DEBUG: Launched block 286 on executor multi with job ID 16896689 parsl.executors.status_handling INFO: Allocated block ID 287 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-287 2024-03-07 15:09:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-287 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-287 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 231 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 231 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=287 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b5ccb290f903f0d27c4ca8acf356abb with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/15abafaf-ecd5-422c-9659-74a3584410c8_isrForCrosstalkSources_1125893_29.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/15abafaf-ecd5-422c-9659-74a3584410c8_isrForCrosstalkSources_1125893_29.stderr parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8a0e8b0cf3d142c4146c16337940b2c with result from task 262 2024-03-07 15:09:06 proc_lsst.multi:170 [INFO] [multi] job_id 16896690 proc_lsst.multi INFO: [multi] job_id 16896690 2024-03-07 15:09:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 232 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 232 2024-03-07 15:09:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896690 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896690 parsl.executors.status_handling DEBUG: Launched block 287 on executor multi with job ID 16896690 parsl.executors.status_handling INFO: Allocated block ID 288 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-288 2024-03-07 15:09:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-288 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-288 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 232 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 232 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=288 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/c6f1291c-d0a4-4364-8615-39091836c6dd_isrForCrosstalkSources_1125905_43.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/c6f1291c-d0a4-4364-8615-39091836c6dd_isrForCrosstalkSources_1125905_43.stderr parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:06 proc_lsst.multi:170 [INFO] [multi] job_id 16896691 proc_lsst.multi INFO: [multi] job_id 16896691 2024-03-07 15:09:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 233 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 233 2024-03-07 15:09:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:06 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896691 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896691 parsl.executors.status_handling DEBUG: Launched block 288 on executor multi with job ID 16896691 parsl.executors.status_handling INFO: Allocated block ID 289 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-289 2024-03-07 15:09:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-289 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-289 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:06 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 233 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 233 2024-03-07 15:09:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=289 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0a9350eeaf6ebe2ae44ff91a7847b41 with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/0412de71-72b8-4040-b53b-cd71aa4cc01a_isrForCrosstalkSources_1125889_58.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/0412de71-72b8-4040-b53b-cd71aa4cc01a_isrForCrosstalkSources_1125889_58.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ecca634d6a66165f76b132a2122b39c with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/bb84629b-f8dd-4b94-b4c9-d3bfb385656a_isrForCrosstalkSources_1125913_42.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/bb84629b-f8dd-4b94-b4c9-d3bfb385656a_isrForCrosstalkSources_1125913_42.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b59e724cfdfac60dcbc75e1feabfb081 with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/666adf21-561a-40ca-ad45-cb5f77dc3bcb_isrForCrosstalkSources_1125930_29.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/666adf21-561a-40ca-ad45-cb5f77dc3bcb_isrForCrosstalkSources_1125930_29.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fc055de21dc1ca5dde54c237f560d0a with result from task 265 2024-03-07 15:09:07 proc_lsst.multi:170 [INFO] [multi] job_id 16896692 proc_lsst.multi INFO: [multi] job_id 16896692 2024-03-07 15:09:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 234 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 234 2024-03-07 15:09:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896692 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896692 parsl.executors.status_handling DEBUG: Launched block 289 on executor multi with job ID 16896692 parsl.executors.status_handling INFO: Allocated block ID 290 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-290 2024-03-07 15:09:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-290 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-290 2024-03-07 15:09:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 234 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 234 2024-03-07 15:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=290 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f138958b-0e82-4564-bcdd-5715508356d2_isrForCrosstalkSources_1125939_44.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f138958b-0e82-4564-bcdd-5715508356d2_isrForCrosstalkSources_1125939_44.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d3c50676a9a6bddfdf8ce9936bd3953 with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/cd7b2219-ea0f-443a-92b3-8d7e186ba01b_isrForCrosstalkSources_1125906_59.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/cd7b2219-ea0f-443a-92b3-8d7e186ba01b_isrForCrosstalkSources_1125906_59.stderr 2024-03-07 15:09:07 proc_lsst.multi:170 [INFO] [multi] job_id 16896693 proc_lsst.multi INFO: [multi] job_id 16896693 2024-03-07 15:09:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 235 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 235 2024-03-07 15:09:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896693 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896693 parsl.executors.status_handling DEBUG: Launched block 290 on executor multi with job ID 16896693 parsl.executors.status_handling INFO: Allocated block ID 291 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-291 2024-03-07 15:09:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-291 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-291 2024-03-07 15:09:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 235 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 235 2024-03-07 15:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=291 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51956aa3e19726e01986e433e739d1be with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/4d52f08f-8321-42e2-ae82-a3364e154af7_isrForCrosstalkSources_1125910_61.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/4d52f08f-8321-42e2-ae82-a3364e154af7_isrForCrosstalkSources_1125910_61.stderr 2024-03-07 15:09:07 proc_lsst.multi:170 [INFO] [multi] job_id 16896694 proc_lsst.multi INFO: [multi] job_id 16896694 2024-03-07 15:09:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 236 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 236 2024-03-07 15:09:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896694 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896694 parsl.executors.status_handling DEBUG: Launched block 291 on executor multi with job ID 16896694 parsl.executors.status_handling INFO: Allocated block ID 292 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-292 2024-03-07 15:09:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-292 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-292 2024-03-07 15:09:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 236 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 236 2024-03-07 15:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=292 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:08 proc_lsst.multi:170 [INFO] [multi] job_id 16896695 proc_lsst.multi INFO: [multi] job_id 16896695 2024-03-07 15:09:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 237 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 237 2024-03-07 15:09:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896695 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896695 parsl.executors.status_handling DEBUG: Launched block 292 on executor multi with job ID 16896695 parsl.executors.status_handling INFO: Allocated block ID 293 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-293 2024-03-07 15:09:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-293 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-293 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 237 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 237 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=293 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61a1dd722c6c03b855faedfdc5114931 with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/2f41f274-a393-47fb-b8e1-54b55698c23b_isrForCrosstalkSources_1125908_62.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/2f41f274-a393-47fb-b8e1-54b55698c23b_isrForCrosstalkSources_1125908_62.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54a2a222a1f0f70930ae789ef4d75741 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/cf31fd17-9857-49e7-98be-022e5af5d2ce_isrForCrosstalkSources_1125918_14.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/cf31fd17-9857-49e7-98be-022e5af5d2ce_isrForCrosstalkSources_1125918_14.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8a14aded3a0571cb471d65d911a1897 with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/3854e754-046c-4f5f-a16f-165a0b16071e_isrForCrosstalkSources_1125894_62.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/3854e754-046c-4f5f-a16f-165a0b16071e_isrForCrosstalkSources_1125894_62.stderr 2024-03-07 15:09:08 proc_lsst.multi:170 [INFO] [multi] job_id 16896696 proc_lsst.multi INFO: [multi] job_id 16896696 2024-03-07 15:09:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 238 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 238 2024-03-07 15:09:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896696 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896696 parsl.executors.status_handling DEBUG: Launched block 293 on executor multi with job ID 16896696 parsl.executors.status_handling INFO: Allocated block ID 294 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-294 2024-03-07 15:09:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-294 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-294 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 238 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 238 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=294 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:08 proc_lsst.multi:170 [INFO] [multi] job_id 16896697 proc_lsst.multi INFO: [multi] job_id 16896697 2024-03-07 15:09:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 239 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 239 2024-03-07 15:09:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896697 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896697 parsl.executors.status_handling DEBUG: Launched block 294 on executor multi with job ID 16896697 parsl.executors.status_handling INFO: Allocated block ID 295 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-295 2024-03-07 15:09:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-295 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-295 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 239 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 239 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=295 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:08 proc_lsst.multi:170 [INFO] [multi] job_id 16896698 proc_lsst.multi INFO: [multi] job_id 16896698 2024-03-07 15:09:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 240 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 240 2024-03-07 15:09:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896698 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896698 parsl.executors.status_handling DEBUG: Launched block 295 on executor multi with job ID 16896698 parsl.executors.status_handling INFO: Allocated block ID 296 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-296 2024-03-07 15:09:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-296 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-296 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 240 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 240 2024-03-07 15:09:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=296 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:09 proc_lsst.multi:170 [INFO] [multi] job_id 16896699 proc_lsst.multi INFO: [multi] job_id 16896699 2024-03-07 15:09:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 241 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 241 2024-03-07 15:09:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896699 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896699 parsl.executors.status_handling DEBUG: Launched block 296 on executor multi with job ID 16896699 parsl.executors.status_handling INFO: Allocated block ID 297 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-297 2024-03-07 15:09:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-297 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-297 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 241 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 241 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=297 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:09 proc_lsst.multi:170 [INFO] [multi] job_id 16896700 proc_lsst.multi INFO: [multi] job_id 16896700 2024-03-07 15:09:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 242 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 242 2024-03-07 15:09:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896700 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896700 parsl.executors.status_handling DEBUG: Launched block 297 on executor multi with job ID 16896700 parsl.executors.status_handling INFO: Allocated block ID 298 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-298 2024-03-07 15:09:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-298 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-298 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 242 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 242 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=298 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf22df2e3530bdfa02ee4aa304e400a8 with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/af1088cf-c36d-4e4a-881b-e769f12c2fc3_isrForCrosstalkSources_1125904_15.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/af1088cf-c36d-4e4a-881b-e769f12c2fc3_isrForCrosstalkSources_1125904_15.stderr 2024-03-07 15:09:09 proc_lsst.multi:170 [INFO] [multi] job_id 16896701 proc_lsst.multi INFO: [multi] job_id 16896701 2024-03-07 15:09:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 243 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 243 2024-03-07 15:09:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896701 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896701 parsl.executors.status_handling DEBUG: Launched block 298 on executor multi with job ID 16896701 parsl.executors.status_handling INFO: Allocated block ID 299 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-299 2024-03-07 15:09:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-299 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-299 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 243 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 243 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=299 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:09 proc_lsst.multi:170 [INFO] [multi] job_id 16896702 proc_lsst.multi INFO: [multi] job_id 16896702 2024-03-07 15:09:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 244 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 244 2024-03-07 15:09:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896702 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896702 parsl.executors.status_handling DEBUG: Launched block 299 on executor multi with job ID 16896702 parsl.executors.status_handling INFO: Allocated block ID 300 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-300 2024-03-07 15:09:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-300 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-300 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 244 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 244 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=300 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:09 proc_lsst.multi:170 [INFO] [multi] job_id 16896703 proc_lsst.multi INFO: [multi] job_id 16896703 2024-03-07 15:09:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 245 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 245 2024-03-07 15:09:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896703 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896703 parsl.executors.status_handling DEBUG: Launched block 300 on executor multi with job ID 16896703 parsl.executors.status_handling INFO: Allocated block ID 301 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-301 2024-03-07 15:09:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-301 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-301 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 245 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 245 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=301 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:09 proc_lsst.multi:170 [INFO] [multi] job_id 16896704 proc_lsst.multi INFO: [multi] job_id 16896704 2024-03-07 15:09:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 246 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 246 2024-03-07 15:09:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896704 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896704 parsl.executors.status_handling DEBUG: Launched block 301 on executor multi with job ID 16896704 parsl.executors.status_handling INFO: Allocated block ID 302 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-302 2024-03-07 15:09:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-302 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-302 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 246 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 246 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=302 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:09 proc_lsst.multi:170 [INFO] [multi] job_id 16896705 proc_lsst.multi INFO: [multi] job_id 16896705 2024-03-07 15:09:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 247 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 247 2024-03-07 15:09:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896705 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896705 parsl.executors.status_handling DEBUG: Launched block 302 on executor multi with job ID 16896705 parsl.executors.status_handling INFO: Allocated block ID 303 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-303 2024-03-07 15:09:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-303 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-303 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 247 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 247 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=303 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f0689e558cb0d345fece4fe6dc1b5ce with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/9ebdaf25-5e4f-4765-bdfd-c95ac7a506ea_isrForCrosstalkSources_1125923_12.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/9ebdaf25-5e4f-4765-bdfd-c95ac7a506ea_isrForCrosstalkSources_1125923_12.stderr 2024-03-07 15:09:09 proc_lsst.multi:170 [INFO] [multi] job_id 16896706 proc_lsst.multi INFO: [multi] job_id 16896706 2024-03-07 15:09:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 248 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 248 2024-03-07 15:09:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896706 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896706 parsl.executors.status_handling DEBUG: Launched block 303 on executor multi with job ID 16896706 parsl.executors.status_handling INFO: Allocated block ID 304 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-304 2024-03-07 15:09:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-304 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-304 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 248 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 248 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=304 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:09 proc_lsst.multi:170 [INFO] [multi] job_id 16896707 proc_lsst.multi INFO: [multi] job_id 16896707 2024-03-07 15:09:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 249 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 249 2024-03-07 15:09:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896707 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896707 parsl.executors.status_handling DEBUG: Launched block 304 on executor multi with job ID 16896707 parsl.executors.status_handling INFO: Allocated block ID 305 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-305 2024-03-07 15:09:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-305 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-305 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 249 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 249 2024-03-07 15:09:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=305 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:10 proc_lsst.multi:170 [INFO] [multi] job_id 16896708 proc_lsst.multi INFO: [multi] job_id 16896708 2024-03-07 15:09:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 250 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 250 2024-03-07 15:09:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896708 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896708 parsl.executors.status_handling DEBUG: Launched block 305 on executor multi with job ID 16896708 parsl.executors.status_handling INFO: Allocated block ID 306 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-306 2024-03-07 15:09:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-306 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-306 2024-03-07 15:09:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 250 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 250 2024-03-07 15:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=306 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:10 proc_lsst.multi:170 [INFO] [multi] job_id 16896709 proc_lsst.multi INFO: [multi] job_id 16896709 2024-03-07 15:09:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 251 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 251 2024-03-07 15:09:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896709 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896709 parsl.executors.status_handling DEBUG: Launched block 306 on executor multi with job ID 16896709 parsl.executors.status_handling INFO: Allocated block ID 307 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-307 2024-03-07 15:09:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-307 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-307 2024-03-07 15:09:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 251 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 251 2024-03-07 15:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=307 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:10 proc_lsst.multi:170 [INFO] [multi] job_id 16896710 proc_lsst.multi INFO: [multi] job_id 16896710 2024-03-07 15:09:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 252 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 252 2024-03-07 15:09:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:10 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896710 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896710 parsl.executors.status_handling DEBUG: Launched block 307 on executor multi with job ID 16896710 parsl.executors.status_handling INFO: Allocated block ID 308 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-308 2024-03-07 15:09:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-308 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-308 2024-03-07 15:09:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:10 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 252 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 252 2024-03-07 15:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=308 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44fb8eda153140ea3b80e369e82802e7 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/0abb1be5-113a-418c-84c2-48efd05115df_isrForCrosstalkSources_1125918_37.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/0abb1be5-113a-418c-84c2-48efd05115df_isrForCrosstalkSources_1125918_37.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68e5a7131402e0358c3eff29ae12a397 with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/01552f8b-a2d3-48aa-9964-c394c983bd06_isrForCrosstalkSources_1125907_51.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/01552f8b-a2d3-48aa-9964-c394c983bd06_isrForCrosstalkSources_1125907_51.stderr 2024-03-07 15:09:11 proc_lsst.multi:170 [INFO] [multi] job_id 16896711 proc_lsst.multi INFO: [multi] job_id 16896711 2024-03-07 15:09:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 253 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 253 2024-03-07 15:09:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896711 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896711 parsl.executors.status_handling DEBUG: Launched block 308 on executor multi with job ID 16896711 parsl.executors.status_handling INFO: Allocated block ID 309 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-309 2024-03-07 15:09:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-309 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-309 2024-03-07 15:09:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 253 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 253 2024-03-07 15:09:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=309 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 15:09:11 proc_lsst.multi:170 [INFO] [multi] job_id 16896712 proc_lsst.multi INFO: [multi] job_id 16896712 2024-03-07 15:09:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 254 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 254 2024-03-07 15:09:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896712 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896712 parsl.executors.status_handling DEBUG: Launched block 309 on executor multi with job ID 16896712 parsl.executors.status_handling INFO: Allocated block ID 310 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-310 2024-03-07 15:09:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-310 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-310 2024-03-07 15:09:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 254 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 254 2024-03-07 15:09:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=310 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00c735df0e568245884900373ead3681 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/c75be3f1-6c97-46a9-bfdf-016b59b88cdd_isrForCrosstalkSources_1125911_55.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/c75be3f1-6c97-46a9-bfdf-016b59b88cdd_isrForCrosstalkSources_1125911_55.stderr 2024-03-07 15:09:11 proc_lsst.multi:170 [INFO] [multi] job_id 16896713 proc_lsst.multi INFO: [multi] job_id 16896713 2024-03-07 15:09:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 255 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 255 2024-03-07 15:09:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:11 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896713 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896713 parsl.executors.status_handling DEBUG: Launched block 310 on executor multi with job ID 16896713 parsl.executors.status_handling INFO: Allocated block ID 311 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-311 2024-03-07 15:09:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-311 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-311 2024-03-07 15:09:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 15:09:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 15:09:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 15:09:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 15:09:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 15:09:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 15:09:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 15:09:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 15:09:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 15:09:11 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 15:09:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 255 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 255 2024-03-07 15:09:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,n3009,169.254.95.120,198.48.92.26,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54581 --result_port=54803 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/runinfo/000/multi --block_id=311 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acd992e7e6e06d3e1e35c84fc1d06f4b with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/d2bb05e0-0711-43a1-85c4-93955c2b051b_isrForCrosstalkSources_1125916_52.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/d2bb05e0-0711-43a1-85c4-93955c2b051b_isrForCrosstalkSources_1125916_52.stderr 2024-03-07 15:09:12 proc_lsst.multi:170 [INFO] [multi] job_id 16896714 proc_lsst.multi INFO: [multi] job_id 16896714 2024-03-07 15:09:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 256 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 256 2024-03-07 15:09:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-07 15:09:12 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16896714 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16896714 parsl.executors.status_handling DEBUG: Launched block 311 on executor multi with job ID 16896714 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22377792571472 2024-03-07 15:09:12 proc_lsst.multi:146 [INFO] found job 53087 in provider local proc_lsst.multi INFO: found job 53087 in provider local 2024-03-07 15:09:12 proc_lsst.multi:146 [INFO] found job 16896385 in provider astro proc_lsst.multi INFO: found job 16896385 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37587f86a922f3831620c9b633ecfb4e with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/3c911783-249c-4786-9d68-6b676ab56601_isrForCrosstalkSources_1125888_7.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/3c911783-249c-4786-9d68-6b676ab56601_isrForCrosstalkSources_1125888_7.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb54157114dc9ee892bec400564a09a9 with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/15fb4bca-209f-47e6-b9e0-45ca3de7831e_isrForCrosstalkSources_1125920_3.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/15fb4bca-209f-47e6-b9e0-45ca3de7831e_isrForCrosstalkSources_1125920_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:12 proc_lsst.multi:146 [INFO] found job 16896386 in provider astro proc_lsst.multi INFO: found job 16896386 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0fb6a4c8677e767beaba6121c82770f with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/a8cb95fa-4925-4251-8878-7c24255f4f8f_isrForCrosstalkSources_1125920_28.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/a8cb95fa-4925-4251-8878-7c24255f4f8f_isrForCrosstalkSources_1125920_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:13 proc_lsst.multi:146 [INFO] found job 16896387 in provider astro proc_lsst.multi INFO: found job 16896387 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5080d887dd0d7289556aa630db6073d1 with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/826ac9c7-e63c-47c0-837a-7f3ce6078437_isrForCrosstalkSources_1125945_15.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/826ac9c7-e63c-47c0-837a-7f3ce6078437_isrForCrosstalkSources_1125945_15.stderr parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d0e0c432a5ab730a630c1bb921b9c3a with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/8d7c1e32-928a-4d4f-b4c4-fca2b62c17ad_isrForCrosstalkSources_1125930_32.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/8d7c1e32-928a-4d4f-b4c4-fca2b62c17ad_isrForCrosstalkSources_1125930_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:13 proc_lsst.multi:146 [INFO] found job 16896388 in provider astro proc_lsst.multi INFO: found job 16896388 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a7101c8d207024eed8dc9a51684895c with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/ccb1a9da-6276-4262-8275-459c3c0a4f10_isrForCrosstalkSources_1125920_40.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/ccb1a9da-6276-4262-8275-459c3c0a4f10_isrForCrosstalkSources_1125920_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:14 proc_lsst.multi:146 [INFO] found job 16896389 in provider astro proc_lsst.multi INFO: found job 16896389 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53db8c39a0f4bab4fe1884c74a549830 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/1c503253-69ad-4790-8825-05e5324ec5c1_isrForCrosstalkSources_1125926_23.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/1c503253-69ad-4790-8825-05e5324ec5c1_isrForCrosstalkSources_1125926_23.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27b8b70c1e9451ac934f65b18053cded with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/e7d37021-b7b0-46c6-b8ce-93b5fad38bf5_isrForCrosstalkSources_1125910_10.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/e7d37021-b7b0-46c6-b8ce-93b5fad38bf5_isrForCrosstalkSources_1125910_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:14 proc_lsst.multi:146 [INFO] found job 16896390 in provider astro proc_lsst.multi INFO: found job 16896390 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9c0974f704f37bf3938f1503a9f3f6d with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/698263a9-86a4-4152-a379-69941e3197b1_isrForCrosstalkSources_1125909_26.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/698263a9-86a4-4152-a379-69941e3197b1_isrForCrosstalkSources_1125909_26.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 167aa4793691c1e82834ece064d31589 with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/e3bb5c17-3402-481c-a0c0-217ca7a08141_isrForCrosstalkSources_1125902_29.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/e3bb5c17-3402-481c-a0c0-217ca7a08141_isrForCrosstalkSources_1125902_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:14 proc_lsst.multi:146 [INFO] found job 16896392 in provider astro proc_lsst.multi INFO: found job 16896392 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b158bd8b9d2026fbf02c5c128d31957 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/2a026a11-9fa9-4b31-a685-3f3551b4a8b0_isrForCrosstalkSources_1125892_43.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/2a026a11-9fa9-4b31-a685-3f3551b4a8b0_isrForCrosstalkSources_1125892_43.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c5f2ef086a4902a88c37fbcae6f604b with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/6f0d18e2-2020-48ae-aa65-47022310fcc0_isrForCrosstalkSources_1125924_57.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/6f0d18e2-2020-48ae-aa65-47022310fcc0_isrForCrosstalkSources_1125924_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:15 proc_lsst.multi:146 [INFO] found job 16896394 in provider astro proc_lsst.multi INFO: found job 16896394 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:15 proc_lsst.multi:146 [INFO] found job 16896395 in provider astro proc_lsst.multi INFO: found job 16896395 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:16 proc_lsst.multi:146 [INFO] found job 16896396 in provider astro proc_lsst.multi INFO: found job 16896396 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:16 proc_lsst.multi:146 [INFO] found job 16896397 in provider astro proc_lsst.multi INFO: found job 16896397 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:16 proc_lsst.multi:146 [INFO] found job 16896398 in provider astro proc_lsst.multi INFO: found job 16896398 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:17 proc_lsst.multi:146 [INFO] found job 16896399 in provider astro proc_lsst.multi INFO: found job 16896399 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:17 proc_lsst.multi:146 [INFO] found job 16896400 in provider astro proc_lsst.multi INFO: found job 16896400 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:17 proc_lsst.multi:146 [INFO] found job 16896401 in provider astro proc_lsst.multi INFO: found job 16896401 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a0f64e38cb206383aa22e33d2f315ad with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/69c42ab2-fa69-4fda-b4ec-874d1caf6376_isrForCrosstalkSources_1125912_14.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/69c42ab2-fa69-4fda-b4ec-874d1caf6376_isrForCrosstalkSources_1125912_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:18 proc_lsst.multi:146 [INFO] found job 16896402 in provider astro proc_lsst.multi INFO: found job 16896402 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab2a219e04112f753fa2b377160605f8 with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/355c6df0-0e71-4c70-a37a-71888d0fdeed_isrForCrosstalkSources_1125916_40.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/355c6df0-0e71-4c70-a37a-71888d0fdeed_isrForCrosstalkSources_1125916_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:18 proc_lsst.multi:146 [INFO] found job 16896403 in provider astro proc_lsst.multi INFO: found job 16896403 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd5b61f82652886690c76923c3c9683b with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/a0667ead-71d8-4c0e-b6ae-6470e6a08150_isrForCrosstalkSources_1125947_7.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/a0667ead-71d8-4c0e-b6ae-6470e6a08150_isrForCrosstalkSources_1125947_7.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d66f06f6efb3c310264b82768037d77 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/f76b57a9-7371-4441-b206-4f47022aff9d_isrForCrosstalkSources_1125929_11.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/f76b57a9-7371-4441-b206-4f47022aff9d_isrForCrosstalkSources_1125929_11.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8b474648b3a0e98ac0427ee36e8eeb5 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/469ffaf9-47c3-48c6-b693-12411826bba0_isrForCrosstalkSources_1125940_56.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/469ffaf9-47c3-48c6-b693-12411826bba0_isrForCrosstalkSources_1125940_56.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 688672dfa957a896da861e20fd229db4 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/c2d1416b-c86d-45c0-97e7-c6ac5dd2d3b7_isrForCrosstalkSources_1125925_48.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/c2d1416b-c86d-45c0-97e7-c6ac5dd2d3b7_isrForCrosstalkSources_1125925_48.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7615302f3a2279ca832f1a6df4c1898 with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/366a27dc-c939-46ed-89f1-3b26405fca74_isrForCrosstalkSources_1125909_46.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/366a27dc-c939-46ed-89f1-3b26405fca74_isrForCrosstalkSources_1125909_46.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9764454b0ce84319645c909833180e18 with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/88ba8e88-e291-4a53-be42-546245aeaf2e_isrForCrosstalkSources_1125912_45.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/88ba8e88-e291-4a53-be42-546245aeaf2e_isrForCrosstalkSources_1125912_45.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c69d08c5373250c5e183d03e7d11a4b8 with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/9fb8f3f4-6b30-4657-9e9c-ca715369af0c_isrForCrosstalkSources_1125947_18.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/9fb8f3f4-6b30-4657-9e9c-ca715369af0c_isrForCrosstalkSources_1125947_18.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be4f26bc53b20ec7995e9301c06ad182 with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/3705d228-d240-4bf6-982c-fc43ed5aa233_isrForCrosstalkSources_1125927_24.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/3705d228-d240-4bf6-982c-fc43ed5aa233_isrForCrosstalkSources_1125927_24.stderr parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32c5e4bf3210dd3aa47ae397295abf57 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/cd410c97-5f89-45de-a111-7858bd4c1357_isrForCrosstalkSources_1125931_23.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/cd410c97-5f89-45de-a111-7858bd4c1357_isrForCrosstalkSources_1125931_23.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4391ab60c41db4091e73e0127b435cf3 with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/85f506b7-e359-4613-aac0-cfcf856bb1ba_isrForCrosstalkSources_1125938_22.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/85f506b7-e359-4613-aac0-cfcf856bb1ba_isrForCrosstalkSources_1125938_22.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6180ad2ba0ea43967278512a91ac4822 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/61f6ae8a-c2f1-4cb8-8ea6-21d7f1d7dc9f_isrForCrosstalkSources_1125944_34.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/61f6ae8a-c2f1-4cb8-8ea6-21d7f1d7dc9f_isrForCrosstalkSources_1125944_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:19 proc_lsst.multi:146 [INFO] found job 16896404 in provider astro proc_lsst.multi INFO: found job 16896404 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14d410712a6a2cb0e1d67e1ef8c93535 with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/7986f352-8146-456f-9fc8-61581f1516dd_isrForCrosstalkSources_1125925_55.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/7986f352-8146-456f-9fc8-61581f1516dd_isrForCrosstalkSources_1125925_55.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7533e987c4cc7fd39a027a97641af906 with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/9a5a5a71-3420-47d5-8984-012cea19a398_isrForCrosstalkSources_1125889_51.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/9a5a5a71-3420-47d5-8984-012cea19a398_isrForCrosstalkSources_1125889_51.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cffbc6cd89ef776bd6fcdd2fe393a625 with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/56fc3d9c-ac57-45ec-ae7c-db6fc9271302_isrForCrosstalkSources_1125896_18.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/56fc3d9c-ac57-45ec-ae7c-db6fc9271302_isrForCrosstalkSources_1125896_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:19 proc_lsst.multi:146 [INFO] found job 16896405 in provider astro proc_lsst.multi INFO: found job 16896405 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c1e5dda81314afa05e88b29002b2f0a with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/ea3ee25b-2ef5-4cc0-aab8-f91800da7c34_isrForCrosstalkSources_1125904_10.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/ea3ee25b-2ef5-4cc0-aab8-f91800da7c34_isrForCrosstalkSources_1125904_10.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2eac40d4719c947cd7786a478e121f92 with result from task 310 parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:19 proc_lsst.multi:146 [INFO] found job 16896408 in provider astro proc_lsst.multi INFO: found job 16896408 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/a554d517-599c-4c1c-86f7-32ea9bce6185_isrForCrosstalkSources_1125903_27.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/a554d517-599c-4c1c-86f7-32ea9bce6185_isrForCrosstalkSources_1125903_27.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97ee712b347c12d95e6684596d0b1032 with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/2b4917aa-26d9-4ae0-a8fa-0b57732e0b5d_isrForCrosstalkSources_1125932_1.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/2b4917aa-26d9-4ae0-a8fa-0b57732e0b5d_isrForCrosstalkSources_1125932_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:20 proc_lsst.multi:146 [INFO] found job 16896409 in provider astro proc_lsst.multi INFO: found job 16896409 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:20 proc_lsst.multi:146 [INFO] found job 16896410 in provider astro proc_lsst.multi INFO: found job 16896410 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fac423ff93fc08e3c59973c0d4d89beb with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/ba06c33e-6b8c-4fc9-982f-7b4f6491c3f0_isrForCrosstalkSources_1125915_47.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/ba06c33e-6b8c-4fc9-982f-7b4f6491c3f0_isrForCrosstalkSources_1125915_47.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c4ff199841474961eb6de88783d5adb with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/c1696cfb-89ed-4972-b6f3-c5b0bb60d6ca_isrForCrosstalkSources_1125890_3.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/c1696cfb-89ed-4972-b6f3-c5b0bb60d6ca_isrForCrosstalkSources_1125890_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:20 proc_lsst.multi:146 [INFO] found job 16896411 in provider astro proc_lsst.multi INFO: found job 16896411 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21d93193ab4a36ded19ab6aee1ba645c with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/ca30b558-0a71-41d9-b3ef-dbe1ba303381_isrForCrosstalkSources_1125946_55.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/ca30b558-0a71-41d9-b3ef-dbe1ba303381_isrForCrosstalkSources_1125946_55.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99b13eca51dc371a8f8a15adfc6b4bf5 with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/3458dc8d-6502-4373-92b0-f8a219be66ea_isrForCrosstalkSources_1125888_62.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/3458dc8d-6502-4373-92b0-f8a219be66ea_isrForCrosstalkSources_1125888_62.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84e41e318c73339a4185932ec762631e with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/beed4d6a-efbc-4879-b909-9b5e676abe60_isrForCrosstalkSources_1125937_54.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/beed4d6a-efbc-4879-b909-9b5e676abe60_isrForCrosstalkSources_1125937_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:20 proc_lsst.multi:146 [INFO] found job 16896412 in provider astro proc_lsst.multi INFO: found job 16896412 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2806cd6a7ca63927a2f1645eb2788c67 with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/14a91392-0e3b-4440-97ac-44c10c1b8083_isrForCrosstalkSources_1125898_59.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/14a91392-0e3b-4440-97ac-44c10c1b8083_isrForCrosstalkSources_1125898_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:21 proc_lsst.multi:146 [INFO] found job 16896413 in provider astro proc_lsst.multi INFO: found job 16896413 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e0b5b25e353817c59deb04b925cdc72 with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/4fc298f7-8e75-4681-8211-b1a75387ae5c_isrForCrosstalkSources_1125894_50.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/4fc298f7-8e75-4681-8211-b1a75387ae5c_isrForCrosstalkSources_1125894_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:21 proc_lsst.multi:146 [INFO] found job 16896414 in provider astro proc_lsst.multi INFO: found job 16896414 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f30ae3569fabc2f18da065a1af33f97 with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/b9ded6a8-c3a0-47aa-b4de-56edf6e784ed_isrForCrosstalkSources_1125899_61.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/b9ded6a8-c3a0-47aa-b4de-56edf6e784ed_isrForCrosstalkSources_1125899_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:21 proc_lsst.multi:146 [INFO] found job 16896415 in provider astro proc_lsst.multi INFO: found job 16896415 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:22 proc_lsst.multi:146 [INFO] found job 16896416 in provider astro proc_lsst.multi INFO: found job 16896416 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:22 proc_lsst.multi:146 [INFO] found job 16896417 in provider astro proc_lsst.multi INFO: found job 16896417 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9810fed7d564e7669563f4816806ed26 with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/2e9837d2-f951-4e56-afa9-9557614c8fd5_isrForCrosstalkSources_1125891_26.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/2e9837d2-f951-4e56-afa9-9557614c8fd5_isrForCrosstalkSources_1125891_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:23 proc_lsst.multi:146 [INFO] found job 16896418 in provider astro proc_lsst.multi INFO: found job 16896418 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d2468c54cbbc122434fbffd7869d6dc with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/3bbe0d0a-5266-455a-ad16-31e7ac2af7ba_isrForCrosstalkSources_1125899_13.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/3bbe0d0a-5266-455a-ad16-31e7ac2af7ba_isrForCrosstalkSources_1125899_13.stderr parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edc3a1fa6994b6711fa584b6fe2a8ac9 with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/68579b87-737d-4375-b0da-461edbb846d4_isrForCrosstalkSources_1125912_33.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/68579b87-737d-4375-b0da-461edbb846d4_isrForCrosstalkSources_1125912_33.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be3a8fa0b6dfa1485c005042dc497891 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/f67ddc4b-c7cc-4407-a359-9284b3a92228_isrForCrosstalkSources_1125926_39.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/f67ddc4b-c7cc-4407-a359-9284b3a92228_isrForCrosstalkSources_1125926_39.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7602c253cf9ad4431a642431f9e1b486 with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7c56235d-f73a-4804-ae72-60d791ed8253_isrForCrosstalkSources_1125929_51.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7c56235d-f73a-4804-ae72-60d791ed8253_isrForCrosstalkSources_1125929_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:23 proc_lsst.multi:146 [INFO] found job 16896419 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896419 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a89568dd4b9e8e5f1fbfa678242301a with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/7025fb9d-e392-4ee4-93a9-968dbc1a2b5c_isrForCrosstalkSources_1125936_21.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/7025fb9d-e392-4ee4-93a9-968dbc1a2b5c_isrForCrosstalkSources_1125936_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:24 proc_lsst.multi:146 [INFO] found job 16896420 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896420 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d33afa27997fa87e206eaaa6b4db54b7 with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/dfddd713-fc13-4a29-aa9e-0e4223077122_isrForCrosstalkSources_1125943_34.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/dfddd713-fc13-4a29-aa9e-0e4223077122_isrForCrosstalkSources_1125943_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:24 proc_lsst.multi:146 [INFO] found job 16896421 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896421 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fd839d015a6176ef9c136ac65909719 with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/2bc976ad-c093-4887-9ba2-ab0ef6a45e67_isrForCrosstalkSources_1125927_2.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/2bc976ad-c093-4887-9ba2-ab0ef6a45e67_isrForCrosstalkSources_1125927_2.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b581f06fbbfffc716765b9c9aed9106 with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/ce27b0a0-5131-4300-ad7d-8134bfdcaa6c_isrForCrosstalkSources_1125927_5.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/ce27b0a0-5131-4300-ad7d-8134bfdcaa6c_isrForCrosstalkSources_1125927_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:24 proc_lsst.multi:146 [INFO] found job 16896422 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896422 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:25 proc_lsst.multi:146 [INFO] found job 16896423 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896423 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:25 proc_lsst.multi:146 [INFO] found job 16896424 in provider escience_a40 proc_lsst.multi INFO: found job 16896424 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4b25122cccb34e0a8b514adff29de53 with result from task 307 parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:25 proc_lsst.multi:146 [INFO] found job 16896425 in provider escience_a40 proc_lsst.multi INFO: found job 16896425 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/1f18e146-c162-4bfd-accf-8c6c6d748298_isrForCrosstalkSources_1125947_33.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/1f18e146-c162-4bfd-accf-8c6c6d748298_isrForCrosstalkSources_1125947_33.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d4af41e5ea5c8f825f347604a01efd3 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/d5bd26bd-e3c8-498e-b4e7-1f56d0aff5f4_isrForCrosstalkSources_1125931_25.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/d5bd26bd-e3c8-498e-b4e7-1f56d0aff5f4_isrForCrosstalkSources_1125931_25.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bcb4c5fee105dd65a86a43a50c3c1fc with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/16db3937-bede-412f-8ded-ac443020390a_isrForCrosstalkSources_1125909_35.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/16db3937-bede-412f-8ded-ac443020390a_isrForCrosstalkSources_1125909_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:26 proc_lsst.multi:146 [INFO] found job 16896426 in provider escience_a40 proc_lsst.multi INFO: found job 16896426 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e34f8e336b6edf5eb6c719fbde96537 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/d0bff332-717c-487d-826b-e8c9a6acfe87_isrForCrosstalkSources_1125909_59.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/d0bff332-717c-487d-826b-e8c9a6acfe87_isrForCrosstalkSources_1125909_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:26 proc_lsst.multi:146 [INFO] found job 16896427 in provider escience_a40 proc_lsst.multi INFO: found job 16896427 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37e2a1d64b2d7ef000285c33fd81f09d with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/d730323a-6051-4b50-b394-36340b9a47c8_isrForCrosstalkSources_1125944_12.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/d730323a-6051-4b50-b394-36340b9a47c8_isrForCrosstalkSources_1125944_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:26 proc_lsst.multi:146 [INFO] found job 16896428 in provider escience_a40 proc_lsst.multi INFO: found job 16896428 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:27 proc_lsst.multi:146 [INFO] found job 16896429 in provider escience_a40 proc_lsst.multi INFO: found job 16896429 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0704dcaca649aabcfede1c35425d5ece with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/6d216ce8-7ff8-4e92-aac5-1f7f62dacdde_isrForCrosstalkSources_1125930_30.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/6d216ce8-7ff8-4e92-aac5-1f7f62dacdde_isrForCrosstalkSources_1125930_30.stderr parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbe4b4b6c6ec79dae32b9ffcbd44f408 with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/523fd886-4c55-4160-b767-10f516889799_isrForCrosstalkSources_1125923_26.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/523fd886-4c55-4160-b767-10f516889799_isrForCrosstalkSources_1125923_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:27 proc_lsst.multi:146 [INFO] found job 16896430 in provider escience_a40 proc_lsst.multi INFO: found job 16896430 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7b5c83e09b3cf4b29d952b2ef24a828 with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/8644772b-8920-4091-a02a-cdb5e0e1ddf7_isrForCrosstalkSources_1125922_28.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/8644772b-8920-4091-a02a-cdb5e0e1ddf7_isrForCrosstalkSources_1125922_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:27 proc_lsst.multi:146 [INFO] found job 16896431 in provider escience_a40 proc_lsst.multi INFO: found job 16896431 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:27 proc_lsst.multi:146 [INFO] found job 16896432 in provider escience_a40 proc_lsst.multi INFO: found job 16896432 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff1e281e83f608780dbe82082ea5213a with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d9687063-7999-47a6-b7d3-b1e35c182557_isrForCrosstalkSources_1125934_4.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d9687063-7999-47a6-b7d3-b1e35c182557_isrForCrosstalkSources_1125934_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:28 proc_lsst.multi:146 [INFO] found job 16896433 in provider escience_a40 proc_lsst.multi INFO: found job 16896433 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce4c68344ff36307be5f165b9443e73d with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/dd713626-1130-406e-b555-a62ff24aff09_isrForCrosstalkSources_1125944_15.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/dd713626-1130-406e-b555-a62ff24aff09_isrForCrosstalkSources_1125944_15.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc9a05a1add02a112822aaf282e8ded1 with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/7119a57d-406b-4557-9639-0bd94f679800_isrForCrosstalkSources_1125919_6.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/7119a57d-406b-4557-9639-0bd94f679800_isrForCrosstalkSources_1125919_6.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a9b44561214c0753b358199e8a03403 with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/c8e86578-fd0f-4001-91eb-7c05e490e280_isrForCrosstalkSources_1125895_3.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/c8e86578-fd0f-4001-91eb-7c05e490e280_isrForCrosstalkSources_1125895_3.stderr parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac2dc039b87b788396afd03406a5c7bd with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/3a324b9b-7747-4fee-9c36-1e58b2a98338_isrForCrosstalkSources_1125930_53.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/3a324b9b-7747-4fee-9c36-1e58b2a98338_isrForCrosstalkSources_1125930_53.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4da48c37389cafc062899edfd4acd8c with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/23308920-2a8f-450b-a24e-62a20a0eda5a_isrForCrosstalkSources_1125889_41.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/23308920-2a8f-450b-a24e-62a20a0eda5a_isrForCrosstalkSources_1125889_41.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 431fa432747336cc32a6a51485b62b1b with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/e2fd778e-e8b8-47a5-b705-1c225f76054d_isrForCrosstalkSources_1125910_31.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/e2fd778e-e8b8-47a5-b705-1c225f76054d_isrForCrosstalkSources_1125910_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:28 proc_lsst.multi:146 [INFO] found job 16896434 in provider escience_a40 proc_lsst.multi INFO: found job 16896434 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66d1ab568232917b5b9c25ab5e21dd83 with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/eed76df3-448f-429d-a318-e51840514d9b_isrForCrosstalkSources_1125937_59.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/eed76df3-448f-429d-a318-e51840514d9b_isrForCrosstalkSources_1125937_59.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9274a7191e5b5d4c394426c2c7bdbee3 with result from task 346 parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:29 proc_lsst.multi:146 [INFO] found job 16896435 in provider escience_a40 proc_lsst.multi INFO: found job 16896435 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/6b6a8fd5-a4f7-496c-8401-9c4d85cd3767_isrForCrosstalkSources_1125898_17.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/6b6a8fd5-a4f7-496c-8401-9c4d85cd3767_isrForCrosstalkSources_1125898_17.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68b6261d061508caeccde77cd1ee6890 with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/cbe82e7c-9899-4d37-9411-12d124ddf8f4_isrForCrosstalkSources_1125913_46.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/cbe82e7c-9899-4d37-9411-12d124ddf8f4_isrForCrosstalkSources_1125913_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:29 proc_lsst.multi:146 [INFO] found job 16896437 in provider escience_a40 proc_lsst.multi INFO: found job 16896437 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:30 proc_lsst.multi:146 [INFO] found job 16896438 in provider escience_a40 proc_lsst.multi INFO: found job 16896438 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:30 proc_lsst.multi:146 [INFO] found job 16896439 in provider escience_a40 proc_lsst.multi INFO: found job 16896439 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:30 proc_lsst.multi:146 [INFO] found job 16896440 in provider escience_a40 proc_lsst.multi INFO: found job 16896440 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:30 proc_lsst.multi:146 [INFO] found job 16896441 in provider escience_a40 proc_lsst.multi INFO: found job 16896441 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:31 proc_lsst.multi:146 [INFO] found job 16896442 in provider escience_a40 proc_lsst.multi INFO: found job 16896442 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:31 proc_lsst.multi:146 [INFO] found job 16896443 in provider escience_a40 proc_lsst.multi INFO: found job 16896443 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c922feb61b5918b6ba781352cbd2d00 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/654405c9-4952-4c6b-8c11-5f2771cbff3b_isrForCrosstalkSources_1125892_27.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/654405c9-4952-4c6b-8c11-5f2771cbff3b_isrForCrosstalkSources_1125892_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:31 proc_lsst.multi:146 [INFO] found job 16896444 in provider escience_a40 proc_lsst.multi INFO: found job 16896444 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:31 proc_lsst.multi:146 [INFO] found job 16896445 in provider checkpoint proc_lsst.multi INFO: found job 16896445 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 920491c9335ce455c085ecc87f696d0d with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/6d598cfa-7e7a-4c34-9cbc-5bfa99b600f1_isrForCrosstalkSources_1125920_13.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/6d598cfa-7e7a-4c34-9cbc-5bfa99b600f1_isrForCrosstalkSources_1125920_13.stderr parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30df9059112dd4c0fa8b9c0b1a1e8a62 with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/79638e92-6ef0-4c2e-92db-c02312d9c133_isrForCrosstalkSources_1125927_20.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/79638e92-6ef0-4c2e-92db-c02312d9c133_isrForCrosstalkSources_1125927_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:31 proc_lsst.multi:146 [INFO] found job 16896446 in provider checkpoint proc_lsst.multi INFO: found job 16896446 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4e27623bd7e8697aafa83ebde682052 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/6d22c9bf-a9b1-4920-ad28-8223236799cf_isrForCrosstalkSources_1125932_26.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/6d22c9bf-a9b1-4920-ad28-8223236799cf_isrForCrosstalkSources_1125932_26.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffea652a04efb51ee62ec10bfaa3f733 with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/8d5bbef2-d998-49fa-8544-d1a3d18a72ed_isrForCrosstalkSources_1125925_45.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/8d5bbef2-d998-49fa-8544-d1a3d18a72ed_isrForCrosstalkSources_1125925_45.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6b4e66b15b5c4763d8ac29a3c5d6dfc with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/5440665e-1b8c-44f9-8313-c2b34e619ae8_isrForCrosstalkSources_1125902_45.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/5440665e-1b8c-44f9-8313-c2b34e619ae8_isrForCrosstalkSources_1125902_45.stderr parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 715e668e35ff137b46078985661f131a with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/40a730c3-6c3a-4bed-90ec-03bb5d082366_isrForCrosstalkSources_1125942_15.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/40a730c3-6c3a-4bed-90ec-03bb5d082366_isrForCrosstalkSources_1125942_15.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02ac7c31d89d3126855746f24f750342 with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/e7b1cd6f-bad3-48d0-86fc-7609f097bc32_isrForCrosstalkSources_1125911_20.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/e7b1cd6f-bad3-48d0-86fc-7609f097bc32_isrForCrosstalkSources_1125911_20.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28e4dde88d68dbd50c0dcee2da89e603 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/e65e7077-8aa7-4cfa-8e65-5958e30a7adb_isrForCrosstalkSources_1125931_50.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/e65e7077-8aa7-4cfa-8e65-5958e30a7adb_isrForCrosstalkSources_1125931_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:32 proc_lsst.multi:146 [INFO] found job 16896447 in provider checkpoint proc_lsst.multi INFO: found job 16896447 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 973c02c1cda3a2ad4e11709363be72a6 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/95af21a9-fe6a-49b7-8529-81b07649d511_isrForCrosstalkSources_1125897_13.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/95af21a9-fe6a-49b7-8529-81b07649d511_isrForCrosstalkSources_1125897_13.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eaae787feeb1d4f348f9dff498952fc3 with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/01392e8e-be9d-47ca-98f9-30e4536ba49d_isrForCrosstalkSources_1125918_41.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/01392e8e-be9d-47ca-98f9-30e4536ba49d_isrForCrosstalkSources_1125918_41.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry becaf2a93f0d4998e086a485c53d3e29 with result from task 347 parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:33 proc_lsst.multi:146 [INFO] found job 16896448 in provider checkpoint proc_lsst.multi INFO: found job 16896448 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/b5d69f2b-bc2b-40b2-8524-c313df353d24_isrForCrosstalkSources_1125891_53.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/b5d69f2b-bc2b-40b2-8524-c313df353d24_isrForCrosstalkSources_1125891_53.stderr parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64f61bed8f95bc90c53568997179943f with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/05d17200-45ea-4160-9a11-2d17c1ec57f1_isrForCrosstalkSources_1125947_14.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/05d17200-45ea-4160-9a11-2d17c1ec57f1_isrForCrosstalkSources_1125947_14.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db66f363bf0c81d8bead17ed9efbc8bf with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/902e0aad-01ac-45f0-ac9c-607f847b7350_isrForCrosstalkSources_1125911_19.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/902e0aad-01ac-45f0-ac9c-607f847b7350_isrForCrosstalkSources_1125911_19.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92486060b0fda475635008989c7c86ae with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/7e95f3c6-2d94-4a34-92eb-b8742fd5712f_isrForCrosstalkSources_1125895_48.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/7e95f3c6-2d94-4a34-92eb-b8742fd5712f_isrForCrosstalkSources_1125895_48.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c1aa7ad8c14b3cfa85691414110e11c with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/ff35a4c2-b774-41ce-ac22-760eb239b416_isrForCrosstalkSources_1125890_33.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/ff35a4c2-b774-41ce-ac22-760eb239b416_isrForCrosstalkSources_1125890_33.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8678a4be08ae04bd9c632d873c001652 with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/a017c7b3-50a2-4af4-9984-cb885403ab0b_isrForCrosstalkSources_1125922_10.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/a017c7b3-50a2-4af4-9984-cb885403ab0b_isrForCrosstalkSources_1125922_10.stderr parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc645bfe32cbe0e523f1719c63a973e4 with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/b8d5e294-091f-42b9-b9ba-25b194e72cf3_isrForCrosstalkSources_1125933_22.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/b8d5e294-091f-42b9-b9ba-25b194e72cf3_isrForCrosstalkSources_1125933_22.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bcaeacb9018ed8536996f781fa9e2cb with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/edf50b99-c192-469a-a11c-6ff783efadbc_isrForCrosstalkSources_1125946_7.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/edf50b99-c192-469a-a11c-6ff783efadbc_isrForCrosstalkSources_1125946_7.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b570b1bb88ffffb3703d1860b5500d4e with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/8c9441c1-0326-4ab8-aea0-ebe88cd60cf8_isrForCrosstalkSources_1125922_42.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/8c9441c1-0326-4ab8-aea0-ebe88cd60cf8_isrForCrosstalkSources_1125922_42.stderr parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a661083174518e5d8d1e1c63aa352f49 with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/72ba23f7-bdfa-46f2-b610-1997f3ebab4a_isrForCrosstalkSources_1125894_18.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/72ba23f7-bdfa-46f2-b610-1997f3ebab4a_isrForCrosstalkSources_1125894_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:35 proc_lsst.multi:146 [INFO] found job 16896449 in provider checkpoint proc_lsst.multi INFO: found job 16896449 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:35 proc_lsst.multi:146 [INFO] found job 16896450 in provider checkpoint proc_lsst.multi INFO: found job 16896450 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4ceb931cb4c7b75c1370917096fe167 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/047eea61-46de-474a-9a66-c85daeb76876_isrForCrosstalkSources_1125896_30.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/047eea61-46de-474a-9a66-c85daeb76876_isrForCrosstalkSources_1125896_30.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 755b42349877ca9edd141b1e7e018505 with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/3b9cda7d-c3bd-459d-90ff-05f86a18f604_isrForCrosstalkSources_1125946_61.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/3b9cda7d-c3bd-459d-90ff-05f86a18f604_isrForCrosstalkSources_1125946_61.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62761bb726a8e316af4a229a42140388 with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/90ab43ac-bd1b-4e39-a509-b5ecc93550e7_isrForCrosstalkSources_1125921_28.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/90ab43ac-bd1b-4e39-a509-b5ecc93550e7_isrForCrosstalkSources_1125921_28.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 950d1eadb08e088c95a648a3e03f050a with result from task 363 parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:36 proc_lsst.multi:146 [INFO] found job 16896451 in provider checkpoint proc_lsst.multi INFO: found job 16896451 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/781d3871-abdd-485e-8f9a-b8f35cdadff7_isrForCrosstalkSources_1125926_41.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/781d3871-abdd-485e-8f9a-b8f35cdadff7_isrForCrosstalkSources_1125926_41.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e9a0ef1096b0f6b4159d58eae199189 with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/8a55f53a-257c-489d-9c4a-664d0e8e1dde_isrForCrosstalkSources_1125928_59.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/8a55f53a-257c-489d-9c4a-664d0e8e1dde_isrForCrosstalkSources_1125928_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:36 proc_lsst.multi:146 [INFO] found job 16896452 in provider checkpoint proc_lsst.multi INFO: found job 16896452 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b31d06e3d5fc6eac878fbac3010516ce with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/606a685e-83cf-4c6d-98ab-577e6f3a3f96_isrForCrosstalkSources_1125918_28.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/606a685e-83cf-4c6d-98ab-577e6f3a3f96_isrForCrosstalkSources_1125918_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:37 proc_lsst.multi:146 [INFO] found job 16896453 in provider checkpoint proc_lsst.multi INFO: found job 16896453 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57157336fd3a1caa7a1efd9bf38db020 with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/f624402e-4183-44e7-b067-a7ddd3a69b91_isrForCrosstalkSources_1125942_19.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/f624402e-4183-44e7-b067-a7ddd3a69b91_isrForCrosstalkSources_1125942_19.stderr parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b140e698f34e69cdd2300039a5e79df with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/8a9c0b13-b389-4e98-be5c-a2242e842e48_isrForCrosstalkSources_1125905_45.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/8a9c0b13-b389-4e98-be5c-a2242e842e48_isrForCrosstalkSources_1125905_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:37 proc_lsst.multi:146 [INFO] found job 16896454 in provider checkpoint proc_lsst.multi INFO: found job 16896454 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a51a972b32f3f8daf459bab2bf5fcc6 with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/51db8048-b99d-4e4b-a2c2-647ebd3df0c5_isrForCrosstalkSources_1125924_22.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/51db8048-b99d-4e4b-a2c2-647ebd3df0c5_isrForCrosstalkSources_1125924_22.stderr parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5539969519b362a1976812aca4b88f1 with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/85bc03a7-cd57-4800-8353-3a25926042de_isrForCrosstalkSources_1125942_58.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/85bc03a7-cd57-4800-8353-3a25926042de_isrForCrosstalkSources_1125942_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:38 proc_lsst.multi:146 [INFO] found job 16896455 in provider checkpoint proc_lsst.multi INFO: found job 16896455 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a2e840247f42c0b27b69b887a75e1d6 with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b9746f55-47da-4e0f-b247-f4692a8d5277_isrForCrosstalkSources_1125938_6.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b9746f55-47da-4e0f-b247-f4692a8d5277_isrForCrosstalkSources_1125938_6.stderr parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d39b9b2c2384fad26848c1482af7e3e7 with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/7b675d7d-f29e-479a-8305-60a62dada638_isrForCrosstalkSources_1125902_36.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/7b675d7d-f29e-479a-8305-60a62dada638_isrForCrosstalkSources_1125902_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:38 proc_lsst.multi:146 [INFO] found job 16896456 in provider checkpoint proc_lsst.multi INFO: found job 16896456 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aea759b4fe97a193848b224805bca95d with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/c63c267b-999c-47e2-8161-877383e1c3d4_isrForCrosstalkSources_1125939_28.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/c63c267b-999c-47e2-8161-877383e1c3d4_isrForCrosstalkSources_1125939_28.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28d80df5131a19f0d7dea8392116cc50 with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/67a6e94c-69e0-418c-878c-c6ac63bc369e_isrForCrosstalkSources_1125906_57.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/67a6e94c-69e0-418c-878c-c6ac63bc369e_isrForCrosstalkSources_1125906_57.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8950f5623cc762bb0f667f09f208ef4 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/43bf59b2-a168-4053-ad49-77084061a06c_isrForCrosstalkSources_1125936_44.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/43bf59b2-a168-4053-ad49-77084061a06c_isrForCrosstalkSources_1125936_44.stderr parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26839d5870f029f24c05914d3e744030 with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/50294325-8a91-4d1b-a685-14cfa41e5a30_isrForCrosstalkSources_1125891_55.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/50294325-8a91-4d1b-a685-14cfa41e5a30_isrForCrosstalkSources_1125891_55.stderr parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45d0766ed1815e9c20a7effd5f54e445 with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/3565cdbe-9a7e-491f-83be-f5ce3b561a22_isrForCrosstalkSources_1125901_60.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/3565cdbe-9a7e-491f-83be-f5ce3b561a22_isrForCrosstalkSources_1125901_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:39 proc_lsst.multi:146 [INFO] found job 16896457 in provider checkpoint proc_lsst.multi INFO: found job 16896457 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:39 proc_lsst.multi:146 [INFO] found job 16896458 in provider checkpoint proc_lsst.multi INFO: found job 16896458 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b47f382d4abad596eb7de4992bca157a with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/e6e72c2d-0b93-476f-93e7-f379e9ce275a_isrForCrosstalkSources_1125892_49.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/e6e72c2d-0b93-476f-93e7-f379e9ce275a_isrForCrosstalkSources_1125892_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:40 proc_lsst.multi:146 [INFO] found job 16896459 in provider checkpoint proc_lsst.multi INFO: found job 16896459 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:40 proc_lsst.multi:146 [INFO] found job 16896460 in provider checkpoint proc_lsst.multi INFO: found job 16896460 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:41 proc_lsst.multi:146 [INFO] found job 16896461 in provider checkpoint proc_lsst.multi INFO: found job 16896461 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f45f658200181742d42c8c2b52e9e2c with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/993ff383-a64a-4851-aa19-f04c833a6088_isrForCrosstalkSources_1125894_5.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/993ff383-a64a-4851-aa19-f04c833a6088_isrForCrosstalkSources_1125894_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:41 proc_lsst.multi:146 [INFO] found job 16896462 in provider checkpoint proc_lsst.multi INFO: found job 16896462 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:41 proc_lsst.multi:146 [INFO] found job 16896463 in provider checkpoint proc_lsst.multi INFO: found job 16896463 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:41 proc_lsst.multi:146 [INFO] found job 16896464 in provider checkpoint proc_lsst.multi INFO: found job 16896464 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0629041c009f02c39b6cd258729a8d8c with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/ccc607d2-ae46-4e18-991d-3eeff7b4324d_isrForCrosstalkSources_1125911_30.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/ccc607d2-ae46-4e18-991d-3eeff7b4324d_isrForCrosstalkSources_1125911_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:41 proc_lsst.multi:146 [INFO] found job 16896466 in provider checkpoint proc_lsst.multi INFO: found job 16896466 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:42 proc_lsst.multi:146 [INFO] found job 16896467 in provider checkpoint proc_lsst.multi INFO: found job 16896467 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cc23312bdd7b359830e024acedf4a2e with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/0c8965b5-de4e-4444-9fcf-4160ee0ee120_isrForCrosstalkSources_1125941_17.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/0c8965b5-de4e-4444-9fcf-4160ee0ee120_isrForCrosstalkSources_1125941_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:42 proc_lsst.multi:146 [INFO] found job 16896468 in provider checkpoint proc_lsst.multi INFO: found job 16896468 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cc4d9e4a81ebbe4c1af358a7f638d7f with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/247b2108-9e82-4689-a91f-5fbae134aef1_isrForCrosstalkSources_1125913_6.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/247b2108-9e82-4689-a91f-5fbae134aef1_isrForCrosstalkSources_1125913_6.stderr parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 681fa744e474781d7d891e9957dee28a with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/5ed561d1-16a1-4464-974b-46551fc0200e_isrForCrosstalkSources_1125890_59.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/5ed561d1-16a1-4464-974b-46551fc0200e_isrForCrosstalkSources_1125890_59.stderr parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e0b98964f4e15c5c3ec23a00a07eac8 with result from task 392 parsl.dataflow.dflow DEBUG: Task 392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/5a184a88-f925-43ea-9324-beda720d23e7_isrForCrosstalkSources_1125910_9.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/5a184a88-f925-43ea-9324-beda720d23e7_isrForCrosstalkSources_1125910_9.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a93b412efbf9a55dfa85841453610d3c with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/88a4d9da-ee8c-4564-9aa9-9ef48fe617c2_isrForCrosstalkSources_1125892_14.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/88a4d9da-ee8c-4564-9aa9-9ef48fe617c2_isrForCrosstalkSources_1125892_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:43 proc_lsst.multi:146 [INFO] found job 16896469 in provider checkpoint proc_lsst.multi INFO: found job 16896469 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6636c435bc06e5aa041c318bb16fafbc with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/98f8abed-977b-4e5f-9449-8731fd6e4fdf_isrForCrosstalkSources_1125907_20.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/98f8abed-977b-4e5f-9449-8731fd6e4fdf_isrForCrosstalkSources_1125907_20.stderr parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fbdb6c2b1c845930fed3ec36adb5caa with result from task 391 parsl.dataflow.dflow DEBUG: Task 391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/1acc40dc-783f-4499-a5f4-55ed89127e7a_isrForCrosstalkSources_1125940_44.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/1acc40dc-783f-4499-a5f4-55ed89127e7a_isrForCrosstalkSources_1125940_44.stderr parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9d5168a18897c9a66a251caa984ddea with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/156e3a55-729d-4646-9275-1bec2e9030d0_isrForCrosstalkSources_1125927_40.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/156e3a55-729d-4646-9275-1bec2e9030d0_isrForCrosstalkSources_1125927_40.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12f9b3acc442f41740cd0eb6f4a163fc with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/bf6525f9-50d7-4bca-85e8-d6306ad16484_isrForCrosstalkSources_1125925_46.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/bf6525f9-50d7-4bca-85e8-d6306ad16484_isrForCrosstalkSources_1125925_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:43 proc_lsst.multi:146 [INFO] found job 16896470 in provider checkpoint proc_lsst.multi INFO: found job 16896470 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efea5190451f6747f66d541c7128fd3e with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/233f97a1-3fd9-4c2a-8ce0-0491d52e01e9_isrForCrosstalkSources_1125907_46.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/233f97a1-3fd9-4c2a-8ce0-0491d52e01e9_isrForCrosstalkSources_1125907_46.stderr parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 500d884f428c06316087f005a0672d8d with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/df66fa8a-7f6f-4394-9560-ddd43ba24685_isrForCrosstalkSources_1125908_41.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/df66fa8a-7f6f-4394-9560-ddd43ba24685_isrForCrosstalkSources_1125908_41.stderr parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53958e529bc190f16a5c83d6030d34f7 with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/493d18ec-f983-4063-a972-3ebcf3ac800b_isrForCrosstalkSources_1125898_23.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/493d18ec-f983-4063-a972-3ebcf3ac800b_isrForCrosstalkSources_1125898_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:44 proc_lsst.multi:146 [INFO] found job 16896471 in provider checkpoint proc_lsst.multi INFO: found job 16896471 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07e41be4256276543b297daa93ee84a0 with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/e337be1e-7f1f-4701-9776-2d571c2b6b19_isrForCrosstalkSources_1125926_32.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/e337be1e-7f1f-4701-9776-2d571c2b6b19_isrForCrosstalkSources_1125926_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:45 proc_lsst.multi:146 [INFO] found job 16896472 in provider checkpoint proc_lsst.multi INFO: found job 16896472 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef5e62b8945404d9e46c0ca236c2be24 with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/6353627f-4d18-4228-a110-8fe2d71dd20c_isrForCrosstalkSources_1125919_25.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/6353627f-4d18-4228-a110-8fe2d71dd20c_isrForCrosstalkSources_1125919_25.stderr parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c613ad7855c7c2d3efeddafeb62c2660 with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/225a948a-3067-49e0-984b-0d26c20ab451_isrForCrosstalkSources_1125930_40.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/225a948a-3067-49e0-984b-0d26c20ab451_isrForCrosstalkSources_1125930_40.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6f90843cef5e76ef4a888ddcfae4db1 with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/daed6dbe-4ea4-4542-a5c2-02b9720498ee_isrForCrosstalkSources_1125910_1.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/daed6dbe-4ea4-4542-a5c2-02b9720498ee_isrForCrosstalkSources_1125910_1.stderr parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e176b01694c90dbac99f0cceccd38777 with result from task 395 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/7e25dec5-3800-42ce-987a-3da04a962cca_isrForCrosstalkSources_1125893_5.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/7e25dec5-3800-42ce-987a-3da04a962cca_isrForCrosstalkSources_1125893_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:45 proc_lsst.multi:146 [INFO] found job 16896473 in provider checkpoint proc_lsst.multi INFO: found job 16896473 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4432d6e172d980635bd55b852712d47e with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/2eb00bba-8ac0-42a6-9c81-00ce2e82ae15_isrForCrosstalkSources_1125925_18.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/2eb00bba-8ac0-42a6-9c81-00ce2e82ae15_isrForCrosstalkSources_1125925_18.stderr parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29a3b7fac08dffa2799de8dc0a514eb9 with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/4514723f-98f8-4c72-b670-086bb897dfc8_isrForCrosstalkSources_1125898_26.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/4514723f-98f8-4c72-b670-086bb897dfc8_isrForCrosstalkSources_1125898_26.stderr parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95ea6465f85a4e2b7c9dfb8329cd2c3b with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/a6a74319-934a-4195-8460-d3b63e157669_isrForCrosstalkSources_1125911_35.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/a6a74319-934a-4195-8460-d3b63e157669_isrForCrosstalkSources_1125911_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:45 proc_lsst.multi:146 [INFO] found job 16896474 in provider checkpoint proc_lsst.multi INFO: found job 16896474 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 622abfcf8ea336c472a9492a3d5a91f3 with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/5d6f4aaf-d480-4c74-9022-bb8da7e0bfa1_isrForCrosstalkSources_1125897_36.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/5d6f4aaf-d480-4c74-9022-bb8da7e0bfa1_isrForCrosstalkSources_1125897_36.stderr parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86847d5139f611c39f543f7dfc9c9a0f with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/6f12ef8d-3dfe-4319-b323-1c737da2aa78_isrForCrosstalkSources_1125896_9.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/6f12ef8d-3dfe-4319-b323-1c737da2aa78_isrForCrosstalkSources_1125896_9.stderr parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99a7b207fa99d8dbd53baeb1e540ae2a with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8aa2aa51-6585-4a0e-b62e-c534dfec92cc_isrForCrosstalkSources_1125936_14.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8aa2aa51-6585-4a0e-b62e-c534dfec92cc_isrForCrosstalkSources_1125936_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:46 proc_lsst.multi:146 [INFO] found job 16896475 in provider checkpoint proc_lsst.multi INFO: found job 16896475 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cfb8f815c2ca0ee4ce96f7dc90427db with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/bc4b82f7-bb06-4758-9ce2-da7986d6487b_isrForCrosstalkSources_1125900_18.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/bc4b82f7-bb06-4758-9ce2-da7986d6487b_isrForCrosstalkSources_1125900_18.stderr parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5a83e405b7b538d6c59748334a63a24 with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/a7b22a2c-4caf-4efa-8c7f-54c20eed8b66_isrForCrosstalkSources_1125922_46.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/a7b22a2c-4caf-4efa-8c7f-54c20eed8b66_isrForCrosstalkSources_1125922_46.stderr parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92c5525b157a8aae7e159c399d449c1b with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/ac9b54f3-7b67-4529-9baf-e85b912d99f1_isrForCrosstalkSources_1125925_42.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/ac9b54f3-7b67-4529-9baf-e85b912d99f1_isrForCrosstalkSources_1125925_42.stderr parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90a9db2bc7c8c615b0d12d0cbe884b34 with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/91d54c1b-d76e-484e-821e-b2d525e340a2_isrForCrosstalkSources_1125909_42.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/91d54c1b-d76e-484e-821e-b2d525e340a2_isrForCrosstalkSources_1125909_42.stderr parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5da51584d2cb8204b6f6811bfdc94f39 with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/031cbf43-48ae-482b-8ff2-15c91296420a_isrForCrosstalkSources_1125894_21.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/031cbf43-48ae-482b-8ff2-15c91296420a_isrForCrosstalkSources_1125894_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:46 proc_lsst.multi:146 [INFO] found job 16896476 in provider checkpoint proc_lsst.multi INFO: found job 16896476 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ccb81437294140907d42a8d414dd1e3 with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/596e536d-c1c2-42ea-b9b5-f3ba2a0c6bcf_isrForCrosstalkSources_1125934_52.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/596e536d-c1c2-42ea-b9b5-f3ba2a0c6bcf_isrForCrosstalkSources_1125934_52.stderr parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ea5497f1957478c66eef222bbc5a0fd with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/b2008e7f-4914-44dc-a889-9a8c9a1d1c31_isrForCrosstalkSources_1125912_44.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/b2008e7f-4914-44dc-a889-9a8c9a1d1c31_isrForCrosstalkSources_1125912_44.stderr parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4bf4e62e260beebbaa36e34c9e0321a with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3c46b709-60bd-4ba7-b5a6-080074f425fb_isrForCrosstalkSources_1125920_55.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3c46b709-60bd-4ba7-b5a6-080074f425fb_isrForCrosstalkSources_1125920_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:47 proc_lsst.multi:146 [INFO] found job 16896477 in provider checkpoint proc_lsst.multi INFO: found job 16896477 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:47 proc_lsst.multi:146 [INFO] found job 16896478 in provider checkpoint proc_lsst.multi INFO: found job 16896478 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10724ad20c040ef1770590b3a23ca3ad with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/a6239eef-ff6f-4397-a098-9038a7b9a69e_isrForCrosstalkSources_1125939_5.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/a6239eef-ff6f-4397-a098-9038a7b9a69e_isrForCrosstalkSources_1125939_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:48 proc_lsst.multi:146 [INFO] found job 16896479 in provider checkpoint proc_lsst.multi INFO: found job 16896479 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec97fb575cf705bda5ac12fd79b64a72 with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/46681dde-bcba-4b84-8383-6c2149542737_isrForCrosstalkSources_1125946_19.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/46681dde-bcba-4b84-8383-6c2149542737_isrForCrosstalkSources_1125946_19.stderr parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 367b6d80d7b82c817eb67dae161d4362 with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/81aa76af-4b67-4c18-bf6d-3dd0e10c229e_isrForCrosstalkSources_1125931_34.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/81aa76af-4b67-4c18-bf6d-3dd0e10c229e_isrForCrosstalkSources_1125931_34.stderr parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aeb6696f2a06874e177f0a2d698abd0a with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/d46cc4ef-6231-4567-b4f8-dfe5aabff808_isrForCrosstalkSources_1125902_52.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/d46cc4ef-6231-4567-b4f8-dfe5aabff808_isrForCrosstalkSources_1125902_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:48 proc_lsst.multi:146 [INFO] found job 16896480 in provider checkpoint proc_lsst.multi INFO: found job 16896480 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 592ea45dce2624b6e60d9e68f2547988 with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/5521f06d-786b-40a3-af40-340fd7787e4c_isrForCrosstalkSources_1125928_17.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/5521f06d-786b-40a3-af40-340fd7787e4c_isrForCrosstalkSources_1125928_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:49 proc_lsst.multi:146 [INFO] found job 16896481 in provider checkpoint proc_lsst.multi INFO: found job 16896481 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 163efa3e33d13621a4c25a92d5dc682e with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/b0a2cdef-b24c-4d9d-953e-7ba7951dd669_isrForCrosstalkSources_1125945_37.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/b0a2cdef-b24c-4d9d-953e-7ba7951dd669_isrForCrosstalkSources_1125945_37.stderr parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9c4d59204a30970521b4d0c2b065460 with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/36b54691-a54a-4038-8e39-d19d8a0fe1b3_isrForCrosstalkSources_1125910_35.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/36b54691-a54a-4038-8e39-d19d8a0fe1b3_isrForCrosstalkSources_1125910_35.stderr parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 971aa4317d1e160cbd6d3831a5890719 with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/ec1e1eff-0d9a-4849-bad4-68a9b5aa60ec_isrForCrosstalkSources_1125932_30.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/ec1e1eff-0d9a-4849-bad4-68a9b5aa60ec_isrForCrosstalkSources_1125932_30.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 422bf65398209aff8474dae6b9b2c6a0 with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/c26b224e-d616-4d0d-86cb-968c0a291ad4_isrForCrosstalkSources_1125924_32.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/c26b224e-d616-4d0d-86cb-968c0a291ad4_isrForCrosstalkSources_1125924_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:50 proc_lsst.multi:146 [INFO] found job 16896482 in provider checkpoint proc_lsst.multi INFO: found job 16896482 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:50 proc_lsst.multi:146 [INFO] found job 16896483 in provider checkpoint proc_lsst.multi INFO: found job 16896483 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:51 proc_lsst.multi:146 [INFO] found job 16896484 in provider checkpoint proc_lsst.multi INFO: found job 16896484 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:51 proc_lsst.multi:146 [INFO] found job 16896485 in provider checkpoint proc_lsst.multi INFO: found job 16896485 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:51 proc_lsst.multi:146 [INFO] found job 16896486 in provider checkpoint proc_lsst.multi INFO: found job 16896486 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:51 proc_lsst.multi:146 [INFO] found job 16896487 in provider checkpoint proc_lsst.multi INFO: found job 16896487 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:52 proc_lsst.multi:146 [INFO] found job 16896488 in provider checkpoint proc_lsst.multi INFO: found job 16896488 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 023027aa52b77f2559a5f2d045d2853f with result from task 416 parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/cbe88845-f718-4162-af62-54b2736e383f_isrForCrosstalkSources_1125908_45.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/cbe88845-f718-4162-af62-54b2736e383f_isrForCrosstalkSources_1125908_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:52 proc_lsst.multi:146 [INFO] found job 16896489 in provider checkpoint proc_lsst.multi INFO: found job 16896489 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 253feb77f79508eaa938f01c0561c774 with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f5a08d8d-3315-436c-864b-3176031007e6_isrForCrosstalkSources_1125939_27.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f5a08d8d-3315-436c-864b-3176031007e6_isrForCrosstalkSources_1125939_27.stderr parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e783a7a41569e43dffaa4006c4eb8aa with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/e9a53ee4-38ff-42ff-ba2e-5c2b57846b65_isrForCrosstalkSources_1125912_36.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/e9a53ee4-38ff-42ff-ba2e-5c2b57846b65_isrForCrosstalkSources_1125912_36.stderr parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 732a81405d34715da3741c9c5faa743e with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/3ef2618e-1ac0-490e-a402-aef4eb1f9b28_isrForCrosstalkSources_1125944_33.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/3ef2618e-1ac0-490e-a402-aef4eb1f9b28_isrForCrosstalkSources_1125944_33.stderr parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35a8fa5737a398b03a8f647d704043fa with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/366d268b-0d8c-4f4e-8b2c-70cd4a6f4909_isrForCrosstalkSources_1125919_14.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/366d268b-0d8c-4f4e-8b2c-70cd4a6f4909_isrForCrosstalkSources_1125919_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:53 proc_lsst.multi:146 [INFO] found job 16896490 in provider checkpoint proc_lsst.multi INFO: found job 16896490 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c588bafbc93932f744ea6e61fe674aa with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/539a2542-721a-41e3-a345-4eceaa73b46c_isrForCrosstalkSources_1125918_47.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/539a2542-721a-41e3-a345-4eceaa73b46c_isrForCrosstalkSources_1125918_47.stderr parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34e0e5fb0bc94132ac32cabe31bece18 with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/8b611a6b-292a-43cc-8418-bcbd4bc08511_isrForCrosstalkSources_1125889_32.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/8b611a6b-292a-43cc-8418-bcbd4bc08511_isrForCrosstalkSources_1125889_32.stderr parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eba169faf11c4df5feff5b29b73ee85e with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/1c3f07c1-fad2-42a1-8e7a-83e6cb6d4e87_isrForCrosstalkSources_1125912_9.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/1c3f07c1-fad2-42a1-8e7a-83e6cb6d4e87_isrForCrosstalkSources_1125912_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:54 proc_lsst.multi:146 [INFO] found job 16896491 in provider checkpoint proc_lsst.multi INFO: found job 16896491 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8fa3a48a09fe369fcf27392b1abd982 with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/a8569b8e-40fd-4896-8821-fa224d46deb2_isrForCrosstalkSources_1125927_21.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/a8569b8e-40fd-4896-8821-fa224d46deb2_isrForCrosstalkSources_1125927_21.stderr parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27ef68946ed1f38e45142781aa1c276a with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f8d1e1b4-b675-4242-a931-4d9c09e09161_isrForCrosstalkSources_1125897_17.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f8d1e1b4-b675-4242-a931-4d9c09e09161_isrForCrosstalkSources_1125897_17.stderr parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bbb1bbddfd28f12c67a0b902cd1ff42 with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/6e25d469-3333-4ea2-82dd-c8e65bebee68_isrForCrosstalkSources_1125913_19.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/6e25d469-3333-4ea2-82dd-c8e65bebee68_isrForCrosstalkSources_1125913_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:54 proc_lsst.multi:146 [INFO] found job 16896492 in provider checkpoint proc_lsst.multi INFO: found job 16896492 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bbc7725d4c91f267d63169256fe7382 with result from task 442 parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/1c42f905-753b-4d97-93e5-9b67c26ff6fd_isrForCrosstalkSources_1125923_25.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/1c42f905-753b-4d97-93e5-9b67c26ff6fd_isrForCrosstalkSources_1125923_25.stderr parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b70202c0cd6edb857cb1dbd512623cd with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/621734b9-9c34-41b5-bbd2-c0650bfc15df_isrForCrosstalkSources_1125934_31.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/621734b9-9c34-41b5-bbd2-c0650bfc15df_isrForCrosstalkSources_1125934_31.stderr parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72df495729545fa270e847dc7ee2501d with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/a2a11641-d8d4-42a2-8547-60f78a81ae5d_isrForCrosstalkSources_1125898_41.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/a2a11641-d8d4-42a2-8547-60f78a81ae5d_isrForCrosstalkSources_1125898_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:55 proc_lsst.multi:146 [INFO] found job 16896493 in provider checkpoint proc_lsst.multi INFO: found job 16896493 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 012117b1c202b8cec3f4ab5511cc197b with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/63361b7a-fe3d-4a9e-90c5-579f61f36e7f_isrForCrosstalkSources_1125901_53.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/63361b7a-fe3d-4a9e-90c5-579f61f36e7f_isrForCrosstalkSources_1125901_53.stderr parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry feb0553034b3766a9a7adb9aae347950 with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/20aa09ba-b0d1-4493-a05f-7b81df107ee9_isrForCrosstalkSources_1125941_38.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/20aa09ba-b0d1-4493-a05f-7b81df107ee9_isrForCrosstalkSources_1125941_38.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79f1a919ed832fa5a68aa29e59dcca48 with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/04ec693a-ce42-4882-b4cd-7c10d2603fa9_isrForCrosstalkSources_1125930_13.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/04ec693a-ce42-4882-b4cd-7c10d2603fa9_isrForCrosstalkSources_1125930_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:55 proc_lsst.multi:146 [INFO] found job 16896494 in provider checkpoint proc_lsst.multi INFO: found job 16896494 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:56 proc_lsst.multi:146 [INFO] found job 16896495 in provider checkpoint proc_lsst.multi INFO: found job 16896495 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 002d5f6239081b426291a28fdd31ce22 with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/14c6834f-fb61-4647-a7c6-1924161f1176_isrForCrosstalkSources_1125902_21.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/14c6834f-fb61-4647-a7c6-1924161f1176_isrForCrosstalkSources_1125902_21.stderr parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4c1ca0ce1fa2ba036d6589e6255a8de with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/f7e4699c-be4b-40ee-bb80-8dad47c35c0e_isrForCrosstalkSources_1125909_40.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/f7e4699c-be4b-40ee-bb80-8dad47c35c0e_isrForCrosstalkSources_1125909_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:56 proc_lsst.multi:146 [INFO] found job 16896496 in provider checkpoint proc_lsst.multi INFO: found job 16896496 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9f43e800ab83569b3f7c578df75022b with result from task 446 parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c25acaaa-588b-4a4a-aab9-ce49e416513f_isrForCrosstalkSources_1125893_21.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c25acaaa-588b-4a4a-aab9-ce49e416513f_isrForCrosstalkSources_1125893_21.stderr parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b18d792643147d3ffc39dcc201d6e1a3 with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4ac203ed-bfef-45cf-9e24-fcfa7e785518_isrForCrosstalkSources_1125893_50.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4ac203ed-bfef-45cf-9e24-fcfa7e785518_isrForCrosstalkSources_1125893_50.stderr parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62195175ab797a505a83db35afad38d0 with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/6e720da9-f615-4ceb-a8cf-034dccd25c52_isrForCrosstalkSources_1125891_52.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/6e720da9-f615-4ceb-a8cf-034dccd25c52_isrForCrosstalkSources_1125891_52.stderr parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f8cac71ca91a8eb1bcee62c25640fb6 with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/95c27012-fb59-4652-a06e-229b77d230f7_isrForCrosstalkSources_1125905_44.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/95c27012-fb59-4652-a06e-229b77d230f7_isrForCrosstalkSources_1125905_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:57 proc_lsst.multi:146 [INFO] found job 16896497 in provider checkpoint proc_lsst.multi INFO: found job 16896497 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4eee2bd9b081800c0a814f130e71a4fe with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/21abffef-4bca-4ef1-ae64-55ec64956c1f_isrForCrosstalkSources_1125891_44.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/21abffef-4bca-4ef1-ae64-55ec64956c1f_isrForCrosstalkSources_1125891_44.stderr parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6810e934b0156d71e41208cd635a0aa4 with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d1aa5469-3487-4e6c-aa75-650536bd1037_isrForCrosstalkSources_1125915_41.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d1aa5469-3487-4e6c-aa75-650536bd1037_isrForCrosstalkSources_1125915_41.stderr parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2aa3e3fc00947367030945343c438a2d with result from task 439 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/022294a5-1fc4-49d3-adf6-8290fa3b1519_isrForCrosstalkSources_1125894_49.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/022294a5-1fc4-49d3-adf6-8290fa3b1519_isrForCrosstalkSources_1125894_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:57 proc_lsst.multi:146 [INFO] found job 16896498 in provider checkpoint proc_lsst.multi INFO: found job 16896498 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d42085be80e6c8101b24201a3619551 with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/6da12762-6447-48fe-a5c8-2dbd19077bb3_isrForCrosstalkSources_1125907_34.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/6da12762-6447-48fe-a5c8-2dbd19077bb3_isrForCrosstalkSources_1125907_34.stderr parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 241437833f28c72a2f26957df12c8828 with result from task 449 parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/f14312a7-fd98-4db2-a286-005ddf68d5f6_isrForCrosstalkSources_1125923_9.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/f14312a7-fd98-4db2-a286-005ddf68d5f6_isrForCrosstalkSources_1125923_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:58 proc_lsst.multi:146 [INFO] found job 16896499 in provider checkpoint proc_lsst.multi INFO: found job 16896499 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c0f8e9bbb3fb6f8746537e96dd13472 with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/93e8928c-485d-4231-b778-2cea8b62b20f_isrForCrosstalkSources_1125915_54.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/93e8928c-485d-4231-b778-2cea8b62b20f_isrForCrosstalkSources_1125915_54.stderr parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8a7b21a74123b1589b535b733bf10f4 with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/301c12ec-5d96-4f98-a068-6904607766e5_isrForCrosstalkSources_1125918_4.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/301c12ec-5d96-4f98-a068-6904607766e5_isrForCrosstalkSources_1125918_4.stderr parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13757561842a481f8d27fcc51d775225 with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/40b394fa-9d01-4272-9f1e-d5e4be7befff_isrForCrosstalkSources_1125913_40.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/40b394fa-9d01-4272-9f1e-d5e4be7befff_isrForCrosstalkSources_1125913_40.stderr parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3096d09d7728b442833578038ac95d4c with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/47c80003-b1dc-4290-8d52-84a9a3e4cf3d_isrForCrosstalkSources_1125898_61.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/47c80003-b1dc-4290-8d52-84a9a3e4cf3d_isrForCrosstalkSources_1125898_61.stderr parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 789c7a10283d9efb588c2cfa640f28a7 with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/1ba0546e-f382-491f-a897-5217be68fac1_isrForCrosstalkSources_1125921_40.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/1ba0546e-f382-491f-a897-5217be68fac1_isrForCrosstalkSources_1125921_40.stderr parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c5372ad2b2d6c2d768e3dd0bcdd407b with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/eaf52586-378c-45b4-a4aa-5e8540cff5cf_isrForCrosstalkSources_1125930_60.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/eaf52586-378c-45b4-a4aa-5e8540cff5cf_isrForCrosstalkSources_1125930_60.stderr parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afb945571529459c90bba5e42ccca65e with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/5119c7dc-cb34-4e0b-924b-2f65f7387369_isrForCrosstalkSources_1125923_59.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/5119c7dc-cb34-4e0b-924b-2f65f7387369_isrForCrosstalkSources_1125923_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:58 proc_lsst.multi:146 [INFO] found job 16896500 in provider checkpoint proc_lsst.multi INFO: found job 16896500 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7256a2b2a0cdeb8e03212cd0786ad590 with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/a4ece2b7-bf06-424d-8a17-c607e7e6f11a_isrForCrosstalkSources_1125930_11.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/a4ece2b7-bf06-424d-8a17-c607e7e6f11a_isrForCrosstalkSources_1125930_11.stderr parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4d849158bf3cc80bb168a4ff1f84abe with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/fcf73a54-5016-4c69-b252-ac8bedf7e052_isrForCrosstalkSources_1125922_37.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/fcf73a54-5016-4c69-b252-ac8bedf7e052_isrForCrosstalkSources_1125922_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:59 proc_lsst.multi:146 [INFO] found job 16896501 in provider checkpoint proc_lsst.multi INFO: found job 16896501 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8f88f62df28a26a5802810e3e110683 with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/8759f061-09bc-493b-b0db-67f79d7fd9e8_isrForCrosstalkSources_1125893_24.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/8759f061-09bc-493b-b0db-67f79d7fd9e8_isrForCrosstalkSources_1125893_24.stderr parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3491a4a9e5d7089878090c4a98d2ad7 with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/41750dd5-33bb-4aa8-b1e8-df7901670e70_isrForCrosstalkSources_1125930_35.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/41750dd5-33bb-4aa8-b1e8-df7901670e70_isrForCrosstalkSources_1125930_35.stderr parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9aaf989161e1343c64428cdf6c53a772 with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/495a7a3c-12e0-4768-a998-f010602dc891_isrForCrosstalkSources_1125907_39.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/495a7a3c-12e0-4768-a998-f010602dc891_isrForCrosstalkSources_1125907_39.stderr parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d4d5dac6c116efa61da7319b0dd5f2b with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/28bf4a89-9ae6-4c92-96e8-1ed4ffba3f43_isrForCrosstalkSources_1125896_60.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/28bf4a89-9ae6-4c92-96e8-1ed4ffba3f43_isrForCrosstalkSources_1125896_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:09:59 proc_lsst.multi:146 [INFO] found job 16896502 in provider checkpoint proc_lsst.multi INFO: found job 16896502 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:00 proc_lsst.multi:146 [INFO] found job 16896503 in provider checkpoint proc_lsst.multi INFO: found job 16896503 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70e581e55f848aa8e74a0487e7e9700d with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/31a2be85-c702-4827-ad13-4babecafc25c_isrForCrosstalkSources_1125889_52.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/31a2be85-c702-4827-ad13-4babecafc25c_isrForCrosstalkSources_1125889_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:00 proc_lsst.multi:146 [INFO] found job 16896504 in provider checkpoint proc_lsst.multi INFO: found job 16896504 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:01 proc_lsst.multi:146 [INFO] found job 16896505 in provider checkpoint proc_lsst.multi INFO: found job 16896505 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:01 proc_lsst.multi:146 [INFO] found job 16896506 in provider checkpoint proc_lsst.multi INFO: found job 16896506 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4ae44f5bdac31f8bed9bb9282f91e8c with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/140176d7-df21-4a37-a608-5d0975c9b234_isrForCrosstalkSources_1125900_49.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/140176d7-df21-4a37-a608-5d0975c9b234_isrForCrosstalkSources_1125900_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:01 proc_lsst.multi:146 [INFO] found job 16896507 in provider checkpoint proc_lsst.multi INFO: found job 16896507 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:02 proc_lsst.multi:146 [INFO] found job 16896508 in provider checkpoint proc_lsst.multi INFO: found job 16896508 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:02 proc_lsst.multi:146 [INFO] found job 16896509 in provider checkpoint proc_lsst.multi INFO: found job 16896509 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98797972456685f00f7ed859b3ea0025 with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/ebf9d239-f0e9-484e-9ddd-4d570461a13e_isrForCrosstalkSources_1125913_11.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/ebf9d239-f0e9-484e-9ddd-4d570461a13e_isrForCrosstalkSources_1125913_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:02 proc_lsst.multi:146 [INFO] found job 16896510 in provider checkpoint proc_lsst.multi INFO: found job 16896510 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:02 proc_lsst.multi:146 [INFO] found job 16896511 in provider checkpoint proc_lsst.multi INFO: found job 16896511 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 514846a73e1b7b89435ffc90b0eb252a with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/22a55563-7821-49ae-b04a-bad6ebe29680_isrForCrosstalkSources_1125906_21.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/22a55563-7821-49ae-b04a-bad6ebe29680_isrForCrosstalkSources_1125906_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:02 proc_lsst.multi:146 [INFO] found job 16896519 in provider checkpoint proc_lsst.multi INFO: found job 16896519 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f1751af7d22817dfc7e12da2416d4d8 with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/06bc21c6-5dab-4c1b-9ec3-8739e4d70325_isrForCrosstalkSources_1125946_46.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/06bc21c6-5dab-4c1b-9ec3-8739e4d70325_isrForCrosstalkSources_1125946_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:03 proc_lsst.multi:146 [INFO] found job 16896520 in provider checkpoint proc_lsst.multi INFO: found job 16896520 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:04 proc_lsst.multi:146 [INFO] found job 16896521 in provider checkpoint proc_lsst.multi INFO: found job 16896521 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:04 proc_lsst.multi:146 [INFO] found job 16896522 in provider checkpoint proc_lsst.multi INFO: found job 16896522 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd9405ddd79dedac2b7914362d299fb5 with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/56058fdd-2950-4664-b6f0-68ecf0de90b4_isrForCrosstalkSources_1125915_3.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/56058fdd-2950-4664-b6f0-68ecf0de90b4_isrForCrosstalkSources_1125915_3.stderr parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5dc8b15253b2aa777486cd80d4696d5 with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/58910a69-8b62-4c88-a54e-03c6014b3f9f_isrForCrosstalkSources_1125906_32.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/58910a69-8b62-4c88-a54e-03c6014b3f9f_isrForCrosstalkSources_1125906_32.stderr parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2364fbd0e62855b17bcadf4adb8761c with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/66bcbcbf-0154-4651-bfff-00bff8af3161_isrForCrosstalkSources_1125909_24.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/66bcbcbf-0154-4651-bfff-00bff8af3161_isrForCrosstalkSources_1125909_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896446 PD 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:05 proc_lsst.multi:146 [INFO] found job 16896523 in provider checkpoint proc_lsst.multi INFO: found job 16896523 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 468f77d37cb2cfff7b84d4b435ab8dcc with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/e73fa079-f05f-484b-8c18-5d01338de168_isrForCrosstalkSources_1125891_15.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/e73fa079-f05f-484b-8c18-5d01338de168_isrForCrosstalkSources_1125891_15.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 136d1646d10ecb073d68c0f8dffe7f45 with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b0ed425e-1a86-4d49-8f9d-04deef0d1998_isrForCrosstalkSources_1125934_14.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b0ed425e-1a86-4d49-8f9d-04deef0d1998_isrForCrosstalkSources_1125934_14.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c5c3bd9f8d9412109479a37d286d500 with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/02cf7d1e-d661-413c-81f0-d84c24927a0c_isrForCrosstalkSources_1125893_36.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/02cf7d1e-d661-413c-81f0-d84c24927a0c_isrForCrosstalkSources_1125893_36.stderr parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3461108ff6f678392c0d3795ee221fdb with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/c5b58a16-bdf3-4b5c-a129-c95f015348f2_isrForCrosstalkSources_1125926_38.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/c5b58a16-bdf3-4b5c-a129-c95f015348f2_isrForCrosstalkSources_1125926_38.stderr parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d92d62b1158fa4d064d71feae19275aa with result from task 463 parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/90852f11-d676-474d-be6b-7079e60ec965_isrForCrosstalkSources_1125895_54.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/90852f11-d676-474d-be6b-7079e60ec965_isrForCrosstalkSources_1125895_54.stderr parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a84e663b09c2ed2e0ff84d6f6f7fc69 with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/35526065-bd2d-4211-a6e3-bfbba08d137d_isrForCrosstalkSources_1125930_50.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/35526065-bd2d-4211-a6e3-bfbba08d137d_isrForCrosstalkSources_1125930_50.stderr parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b7873d2739a1e95e8d3ef81736fa273 with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/349a0efd-3836-4a73-b2f5-124c0421f07d_isrForCrosstalkSources_1125940_22.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/349a0efd-3836-4a73-b2f5-124c0421f07d_isrForCrosstalkSources_1125940_22.stderr parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4413f237b4727604c19f82f7659d945 with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/26588652-1351-4faf-a049-84ce0a2c267e_isrForCrosstalkSources_1125905_7.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/26588652-1351-4faf-a049-84ce0a2c267e_isrForCrosstalkSources_1125905_7.stderr parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1aee264e802e1493d66fbcc0e9d559f6 with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/6602bd34-e0b7-4211-8c6f-1a31093a7352_isrForCrosstalkSources_1125930_3.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/6602bd34-e0b7-4211-8c6f-1a31093a7352_isrForCrosstalkSources_1125930_3.stderr parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28079513379b43018bda2925a03bc5a5 with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/13d4dab4-1995-4a1a-891c-c218fe60133b_isrForCrosstalkSources_1125924_43.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/13d4dab4-1995-4a1a-891c-c218fe60133b_isrForCrosstalkSources_1125924_43.stderr parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 018b9429b39a0279de18d3f14c368824 with result from task 490 parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:07 proc_lsst.multi:146 [INFO] found job 16896524 in provider checkpoint proc_lsst.multi INFO: found job 16896524 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/718054d1-03e9-4e4e-9a4d-2e99f699d1a1_isrForCrosstalkSources_1125896_46.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/718054d1-03e9-4e4e-9a4d-2e99f699d1a1_isrForCrosstalkSources_1125896_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:07 proc_lsst.multi:146 [INFO] found job 16896525 in provider checkpoint proc_lsst.multi INFO: found job 16896525 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bccb0148c5a80772ea99cd88ea0beea0 with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/d68be105-faed-4fa6-af3e-7aa9b1a086cb_isrForCrosstalkSources_1125901_43.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/d68be105-faed-4fa6-af3e-7aa9b1a086cb_isrForCrosstalkSources_1125901_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:08 proc_lsst.multi:146 [INFO] found job 16896526 in provider checkpoint proc_lsst.multi INFO: found job 16896526 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8b615797c622361be9be8e1f56ec5a6 with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/e546d399-6ef4-4440-8f99-73f7d3770200_isrForCrosstalkSources_1125905_6.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/e546d399-6ef4-4440-8f99-73f7d3770200_isrForCrosstalkSources_1125905_6.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5bb952f32d5c5de6fa44c3a6acf9be1 with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/f8dba169-4a64-4351-a8c2-9e218fe90978_isrForCrosstalkSources_1125903_18.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/f8dba169-4a64-4351-a8c2-9e218fe90978_isrForCrosstalkSources_1125903_18.stderr parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b1776cf9e4f6c1b79d97b6922ca3bcb with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3372171c-c045-4389-878b-f09f29eb27e6_isrForCrosstalkSources_1125941_47.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3372171c-c045-4389-878b-f09f29eb27e6_isrForCrosstalkSources_1125941_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:08 proc_lsst.multi:146 [INFO] found job 16896527 in provider checkpoint proc_lsst.multi INFO: found job 16896527 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26e00e2fa255a2882a6c0394c2d7ce5d with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/0db15fde-958f-4372-8ac6-d9759ee3a8ae_isrForCrosstalkSources_1125927_11.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/0db15fde-958f-4372-8ac6-d9759ee3a8ae_isrForCrosstalkSources_1125927_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:09 proc_lsst.multi:146 [INFO] found job 16896528 in provider checkpoint proc_lsst.multi INFO: found job 16896528 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e1aafce9550198b64fbf6376ae5e451 with result from task 501 parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/2826457e-d018-49ee-9887-e349163714c8_isrForCrosstalkSources_1125923_19.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/2826457e-d018-49ee-9887-e349163714c8_isrForCrosstalkSources_1125923_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:10 proc_lsst.multi:146 [INFO] found job 16896529 in provider checkpoint proc_lsst.multi INFO: found job 16896529 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 011465ade036ddbf2b65b61bb51d15fe with result from task 481 parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/6cc12926-b68d-4189-a34a-3360d2ed97bb_isrForCrosstalkSources_1125889_27.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/6cc12926-b68d-4189-a34a-3360d2ed97bb_isrForCrosstalkSources_1125889_27.stderr parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcca4c993ae01778e42067007a09a884 with result from task 503 parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:10 proc_lsst.multi:146 [INFO] found job 16896530 in provider checkpoint proc_lsst.multi INFO: found job 16896530 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4fe26ac8-87b8-4a2c-8f41-cac6578ef2cf_isrForCrosstalkSources_1125908_26.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4fe26ac8-87b8-4a2c-8f41-cac6578ef2cf_isrForCrosstalkSources_1125908_26.stderr parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d661c60e4cf8800b5d2620d3da0d8bc with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/feed14e1-bd72-4de0-98e6-04a9663574e9_isrForCrosstalkSources_1125938_46.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/feed14e1-bd72-4de0-98e6-04a9663574e9_isrForCrosstalkSources_1125938_46.stderr parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 054c7b9479aef1ccb5f8aee602c78ab2 with result from task 482 parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/f64c1663-3b81-44c0-b833-79ec598b3de3_isrForCrosstalkSources_1125915_21.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/f64c1663-3b81-44c0-b833-79ec598b3de3_isrForCrosstalkSources_1125915_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:11 proc_lsst.multi:146 [INFO] found job 16896531 in provider checkpoint proc_lsst.multi INFO: found job 16896531 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:11 proc_lsst.multi:146 [INFO] found job 16896533 in provider checkpoint proc_lsst.multi INFO: found job 16896533 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 148a9721bd49d277156997f0a30e4c22 with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/a2e4a30f-f43b-4b58-b3f1-686875e1893d_isrForCrosstalkSources_1125926_51.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/a2e4a30f-f43b-4b58-b3f1-686875e1893d_isrForCrosstalkSources_1125926_51.stderr parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c834cb02b05317f7e2e5fc2a70bf726 with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/88cac695-4245-4493-bf37-34650be59f71_isrForCrosstalkSources_1125921_18.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/88cac695-4245-4493-bf37-34650be59f71_isrForCrosstalkSources_1125921_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:11 proc_lsst.multi:146 [INFO] found job 16896534 in provider checkpoint proc_lsst.multi INFO: found job 16896534 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d90f6ad0354284e9d49bebb41577659b with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/17809f5b-d724-4a2c-9af7-afd3d21952cb_isrForCrosstalkSources_1125892_6.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/17809f5b-d724-4a2c-9af7-afd3d21952cb_isrForCrosstalkSources_1125892_6.stderr parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 177cec6fa1250047656055b72ee99333 with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/e938b9d1-e2da-4713-849c-124892214b63_isrForCrosstalkSources_1125927_9.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/e938b9d1-e2da-4713-849c-124892214b63_isrForCrosstalkSources_1125927_9.stderr parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a7ed80bdf719a9c8ea4fd0438417acd with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/196e2add-df0b-4b40-81b0-87e76d4c6a40_isrForCrosstalkSources_1125945_11.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/196e2add-df0b-4b40-81b0-87e76d4c6a40_isrForCrosstalkSources_1125945_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:11 proc_lsst.multi:146 [INFO] found job 16896535 in provider checkpoint proc_lsst.multi INFO: found job 16896535 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c38675da7d7478dd0e04fd3431fe1762 with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c1732c9f-5bff-45c7-a934-b110bf4e75bc_isrForCrosstalkSources_1125910_15.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c1732c9f-5bff-45c7-a934-b110bf4e75bc_isrForCrosstalkSources_1125910_15.stderr parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9b788829789d168d04af09ef35954c5 with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/25b2a31c-7ee8-4280-8bbd-96a4a307cfea_isrForCrosstalkSources_1125932_53.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/25b2a31c-7ee8-4280-8bbd-96a4a307cfea_isrForCrosstalkSources_1125932_53.stderr parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a0186b98dd89e3c41ebffcccb817bca with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/086d78e5-1e60-4692-a774-0933cd388f91_isrForCrosstalkSources_1125896_2.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/086d78e5-1e60-4692-a774-0933cd388f91_isrForCrosstalkSources_1125896_2.stderr parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb4e0fe894513631cbecb2326b0dd054 with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/a5dbaa89-a33c-4697-b881-358a66a52f23_isrForCrosstalkSources_1125929_3.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/a5dbaa89-a33c-4697-b881-358a66a52f23_isrForCrosstalkSources_1125929_3.stderr parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c983db0a1b1c7ad299bd2aea988675a6 with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/29f6df08-e30b-48c6-a637-7cab746d7dcf_isrForCrosstalkSources_1125932_38.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/29f6df08-e30b-48c6-a637-7cab746d7dcf_isrForCrosstalkSources_1125932_38.stderr parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 060d76acb69e8139f1976f3d362eb689 with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/82efe159-dfbf-4b53-a4e5-ed67790a4945_isrForCrosstalkSources_1125891_57.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/82efe159-dfbf-4b53-a4e5-ed67790a4945_isrForCrosstalkSources_1125891_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:12 proc_lsst.multi:146 [INFO] found job 16896536 in provider checkpoint proc_lsst.multi INFO: found job 16896536 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a253c57d9e73f7cccab123acb1d30095 with result from task 497 parsl.dataflow.dflow DEBUG: Task 497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/54d7c0de-4684-4305-8008-97fa6db7ec01_isrForCrosstalkSources_1125899_11.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/54d7c0de-4684-4305-8008-97fa6db7ec01_isrForCrosstalkSources_1125899_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:12 proc_lsst.multi:146 [INFO] found job 16896537 in provider checkpoint proc_lsst.multi INFO: found job 16896537 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3519e17395ca9b4200441fa889a9e506 with result from task 472 parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/7b98f1bc-2068-47d4-a7ac-11a19dc88542_isrForCrosstalkSources_1125906_53.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/7b98f1bc-2068-47d4-a7ac-11a19dc88542_isrForCrosstalkSources_1125906_53.stderr parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6127bb6b03ebf71d750669cd5badd931 with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/a143b94a-b461-45bc-81b6-ddd26ba0d609_isrForCrosstalkSources_1125911_57.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/a143b94a-b461-45bc-81b6-ddd26ba0d609_isrForCrosstalkSources_1125911_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:12 proc_lsst.multi:146 [INFO] found job 16896538 in provider checkpoint proc_lsst.multi INFO: found job 16896538 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a440afbd725e5ea9af97339e3e3be260 with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2d37a09b-8b5a-4753-a28a-9f91df1bc949_isrForCrosstalkSources_1125895_62.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2d37a09b-8b5a-4753-a28a-9f91df1bc949_isrForCrosstalkSources_1125895_62.stderr parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 663de4db32ef7369ac0c195ef5f079b6 with result from task 491 parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/f64ba87d-eeea-46fa-a0df-0ebc937b2c22_isrForCrosstalkSources_1125891_54.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/f64ba87d-eeea-46fa-a0df-0ebc937b2c22_isrForCrosstalkSources_1125891_54.stderr parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96deba7b967ac76670ae2236e769fb50 with result from task 500 parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/377ef147-0129-4a72-a4ec-ba990d45d0a3_isrForCrosstalkSources_1125894_41.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/377ef147-0129-4a72-a4ec-ba990d45d0a3_isrForCrosstalkSources_1125894_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:13 proc_lsst.multi:146 [INFO] found job 16896539 in provider checkpoint proc_lsst.multi INFO: found job 16896539 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c894b118778f2d1c4cc7899c6dfa024 with result from task 488 parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/5ed94791-e096-4438-813d-a6f1cebde254_isrForCrosstalkSources_1125918_62.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/5ed94791-e096-4438-813d-a6f1cebde254_isrForCrosstalkSources_1125918_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:14 proc_lsst.multi:146 [INFO] found job 16896540 in provider checkpoint proc_lsst.multi INFO: found job 16896540 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:14 proc_lsst.multi:146 [INFO] found job 16896541 in provider checkpoint proc_lsst.multi INFO: found job 16896541 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 288a89b251aa4322ca8328a1abf79753 with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/3ae108cb-e837-4258-ad7e-701c068e3172_isrForCrosstalkSources_1125937_16.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/3ae108cb-e837-4258-ad7e-701c068e3172_isrForCrosstalkSources_1125937_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:15 proc_lsst.multi:146 [INFO] found job 16896542 in provider checkpoint proc_lsst.multi INFO: found job 16896542 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:15 proc_lsst.multi:146 [INFO] found job 16896543 in provider checkpoint proc_lsst.multi INFO: found job 16896543 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:16 proc_lsst.multi:146 [INFO] found job 16896544 in provider checkpoint proc_lsst.multi INFO: found job 16896544 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff680d63bff55fc95510edd567451c59 with result from task 509 parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/5e6ccfa9-b52f-4fdb-b2cf-38fd9137455b_isrForCrosstalkSources_1125939_43.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/5e6ccfa9-b52f-4fdb-b2cf-38fd9137455b_isrForCrosstalkSources_1125939_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:16 proc_lsst.multi:146 [INFO] found job 16896545 in provider checkpoint proc_lsst.multi INFO: found job 16896545 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3c3a36446b509aa1063c71e92c7de73 with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/df1a5746-c32b-4751-b727-842628cde99c_isrForCrosstalkSources_1125888_43.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/df1a5746-c32b-4751-b727-842628cde99c_isrForCrosstalkSources_1125888_43.stderr parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc6b28c07f9ea0b570a62afde0c7b518 with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/60d450e1-073c-4ab0-8893-63ada0ee4ebe_isrForCrosstalkSources_1125925_43.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/60d450e1-073c-4ab0-8893-63ada0ee4ebe_isrForCrosstalkSources_1125925_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:16 proc_lsst.multi:146 [INFO] found job 16896546 in provider checkpoint proc_lsst.multi INFO: found job 16896546 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:17 proc_lsst.multi:146 [INFO] found job 16896547 in provider checkpoint proc_lsst.multi INFO: found job 16896547 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bbb19dea926c7c6c62a5af107b9bb0e with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/9aa77323-411d-4fd5-9d7e-8a94b28bc577_isrForCrosstalkSources_1125911_52.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/9aa77323-411d-4fd5-9d7e-8a94b28bc577_isrForCrosstalkSources_1125911_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:17 proc_lsst.multi:146 [INFO] found job 16896548 in provider checkpoint proc_lsst.multi INFO: found job 16896548 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b839fc7a60ac042e4b41a24a4d82a0cd with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/4ec6fbe1-2d2e-4762-a17f-915ffc32a93f_isrForCrosstalkSources_1125888_17.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/4ec6fbe1-2d2e-4762-a17f-915ffc32a93f_isrForCrosstalkSources_1125888_17.stderr parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c3be0b8f278013192b5a9631eedc745 with result from task 508 parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/80c826ed-2f80-4279-a5fe-10047729b064_isrForCrosstalkSources_1125930_5.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/80c826ed-2f80-4279-a5fe-10047729b064_isrForCrosstalkSources_1125930_5.stderr parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 270abb54e38b3b5994793db03e4dde56 with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/4c267442-9217-425d-bc85-9715df9e3170_isrForCrosstalkSources_1125888_59.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/4c267442-9217-425d-bc85-9715df9e3170_isrForCrosstalkSources_1125888_59.stderr parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d1bcc8b7821ad1b5e7c562e94240fb6 with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/74fd42d3-512d-4641-995a-8bf790e9923e_isrForCrosstalkSources_1125933_45.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/74fd42d3-512d-4641-995a-8bf790e9923e_isrForCrosstalkSources_1125933_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:17 proc_lsst.multi:146 [INFO] found job 16896549 in provider checkpoint proc_lsst.multi INFO: found job 16896549 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63a5005ef1e73349afcafb7d70534817 with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/f929ad59-51cd-4441-85b4-d82006aa0206_isrForCrosstalkSources_1125931_39.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/f929ad59-51cd-4441-85b4-d82006aa0206_isrForCrosstalkSources_1125931_39.stderr parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dca0ac3f3983882fb2e250d9f6a2bed with result from task 510 parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/9e21793a-66d8-4eb4-bdae-6b9836608e87_isrForCrosstalkSources_1125942_39.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/9e21793a-66d8-4eb4-bdae-6b9836608e87_isrForCrosstalkSources_1125942_39.stderr parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1375bec2ece52d04fe3b809addc3eff9 with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/d385a75e-1907-4dd4-8d6b-566873299594_isrForCrosstalkSources_1125913_51.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/d385a75e-1907-4dd4-8d6b-566873299594_isrForCrosstalkSources_1125913_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:18 proc_lsst.multi:146 [INFO] found job 16896550 in provider checkpoint proc_lsst.multi INFO: found job 16896550 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed54fb8e7edebb9f28362ffd3118c0cd with result from task 524 parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/bff316f1-1c96-42d4-8d0d-54bee473cd9e_isrForCrosstalkSources_1125897_30.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/bff316f1-1c96-42d4-8d0d-54bee473cd9e_isrForCrosstalkSources_1125897_30.stderr parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbc9782e952aeb9175d0be8fbda745c4 with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/25586c3e-46a2-44ab-863e-13ff89307f05_isrForCrosstalkSources_1125917_54.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/25586c3e-46a2-44ab-863e-13ff89307f05_isrForCrosstalkSources_1125917_54.stderr parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bdb7f2f8429e13ac679f710e611abd0 with result from task 523 parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/b4273a2c-9e0f-479b-a5c9-14f95a952767_isrForCrosstalkSources_1125936_16.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/b4273a2c-9e0f-479b-a5c9-14f95a952767_isrForCrosstalkSources_1125936_16.stderr parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 122a478a560f980a972bbb64e3685299 with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/1017e6ac-feaa-42d4-98b4-473b23dc6dc2_isrForCrosstalkSources_1125896_49.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/1017e6ac-feaa-42d4-98b4-473b23dc6dc2_isrForCrosstalkSources_1125896_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:18 proc_lsst.multi:146 [INFO] found job 16896551 in provider checkpoint proc_lsst.multi INFO: found job 16896551 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01699d96c219e35e0184ac9134fa0dd0 with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/95723426-7f49-4cb9-9975-5c8f2d26be70_isrForCrosstalkSources_1125905_50.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/95723426-7f49-4cb9-9975-5c8f2d26be70_isrForCrosstalkSources_1125905_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:19 proc_lsst.multi:146 [INFO] found job 16896552 in provider checkpoint proc_lsst.multi INFO: found job 16896552 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e05556534383ff783c5db208d339d3f8 with result from task 526 parsl.dataflow.dflow DEBUG: Task 526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/25b4f6f0-404b-403d-9ecd-ef42aa06a27e_isrForCrosstalkSources_1125926_37.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/25b4f6f0-404b-403d-9ecd-ef42aa06a27e_isrForCrosstalkSources_1125926_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:19 proc_lsst.multi:146 [INFO] found job 16896553 in provider checkpoint proc_lsst.multi INFO: found job 16896553 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8a25c5c4c7601d2a485a48222aab9c3 with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/e3d7dfb0-5cbc-413e-9e7e-166997869a1f_isrForCrosstalkSources_1125925_35.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/e3d7dfb0-5cbc-413e-9e7e-166997869a1f_isrForCrosstalkSources_1125925_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:19 proc_lsst.multi:146 [INFO] found job 16896554 in provider checkpoint proc_lsst.multi INFO: found job 16896554 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 915ce6c73b387df71e04dc9a7acf7f62 with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/abd322ea-9e2d-4e07-877d-01de1989b3f0_isrForCrosstalkSources_1125931_58.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/abd322ea-9e2d-4e07-877d-01de1989b3f0_isrForCrosstalkSources_1125931_58.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4eded6dbc208dac40347e6a9bdd15f6b with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/3c3ffce1-b3d6-4b14-b9f8-26cb5884cbaf_isrForCrosstalkSources_1125918_49.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/3c3ffce1-b3d6-4b14-b9f8-26cb5884cbaf_isrForCrosstalkSources_1125918_49.stderr parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77c2e2de74f65d6c3ace3acfee9e0271 with result from task 533 parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/25544466-6e68-4742-ae9e-e3b6eed92a6d_isrForCrosstalkSources_1125946_23.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/25544466-6e68-4742-ae9e-e3b6eed92a6d_isrForCrosstalkSources_1125946_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:20 proc_lsst.multi:146 [INFO] found job 16896555 in provider checkpoint proc_lsst.multi INFO: found job 16896555 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:20 proc_lsst.multi:146 [INFO] found job 16896556 in provider checkpoint proc_lsst.multi INFO: found job 16896556 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:21 proc_lsst.multi:146 [INFO] found job 16896559 in provider checkpoint proc_lsst.multi INFO: found job 16896559 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:21 proc_lsst.multi:146 [INFO] found job 16896561 in provider checkpoint proc_lsst.multi INFO: found job 16896561 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6993791a509e18f5e962da2611859d3 with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/3d72ae97-02cc-4126-b263-54076e553b12_isrForCrosstalkSources_1125924_38.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/3d72ae97-02cc-4126-b263-54076e553b12_isrForCrosstalkSources_1125924_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:22 proc_lsst.multi:146 [INFO] found job 16896562 in provider checkpoint proc_lsst.multi INFO: found job 16896562 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:22 proc_lsst.multi:146 [INFO] found job 16896563 in provider checkpoint proc_lsst.multi INFO: found job 16896563 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:22 proc_lsst.multi:146 [INFO] found job 16896564 in provider checkpoint proc_lsst.multi INFO: found job 16896564 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:22 proc_lsst.multi:146 [INFO] found job 16896565 in provider checkpoint proc_lsst.multi INFO: found job 16896565 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:23 proc_lsst.multi:146 [INFO] found job 16896566 in provider checkpoint proc_lsst.multi INFO: found job 16896566 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0a1a2cdf0a962617e64667e5ad425eb with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/a85afe2d-24d9-4e23-9c0e-0aa689aef33a_isrForCrosstalkSources_1125911_34.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/a85afe2d-24d9-4e23-9c0e-0aa689aef33a_isrForCrosstalkSources_1125911_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:23 proc_lsst.multi:146 [INFO] found job 16896568 in provider checkpoint proc_lsst.multi INFO: found job 16896568 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ced9c9df0e57c81d5294a66db783a39d with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/ea61118a-ee15-441a-92a8-5bcf067f237e_isrForCrosstalkSources_1125888_19.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/ea61118a-ee15-441a-92a8-5bcf067f237e_isrForCrosstalkSources_1125888_19.stderr parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 487075abdfefb76d73e3ff397fda1c74 with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/4b38c2ec-4d26-4f15-abfb-70e0f2296bb3_isrForCrosstalkSources_1125901_56.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/4b38c2ec-4d26-4f15-abfb-70e0f2296bb3_isrForCrosstalkSources_1125901_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:23 proc_lsst.multi:146 [INFO] found job 16896569 in provider checkpoint proc_lsst.multi INFO: found job 16896569 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c638ccf61927c2640a83023831039e9 with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/1c8c1283-4cbc-4a04-84e7-610707a9b445_isrForCrosstalkSources_1125946_1.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/1c8c1283-4cbc-4a04-84e7-610707a9b445_isrForCrosstalkSources_1125946_1.stderr parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f8577a7d5377623243cd7d6dd59bec5 with result from task 538 parsl.dataflow.dflow DEBUG: Task 538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/ac91cffe-56c0-4882-bc73-7e1a945e404f_isrForCrosstalkSources_1125944_37.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/ac91cffe-56c0-4882-bc73-7e1a945e404f_isrForCrosstalkSources_1125944_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:24 proc_lsst.multi:146 [INFO] found job 16896570 in provider checkpoint proc_lsst.multi INFO: found job 16896570 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8ffb3d38840b7aa3ca9dde0e9c75af9 with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/446dfe63-a7fa-4a7c-bea3-d95f675e9097_isrForCrosstalkSources_1125888_57.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/446dfe63-a7fa-4a7c-bea3-d95f675e9097_isrForCrosstalkSources_1125888_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:24 proc_lsst.multi:146 [INFO] found job 16896571 in provider checkpoint proc_lsst.multi INFO: found job 16896571 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 706f9ff774ce4bcbdfda7b1a99eb665c with result from task 527 parsl.dataflow.dflow DEBUG: Task 527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/df4e23d1-4e46-4a97-a281-482cdab78f81_isrForCrosstalkSources_1125894_58.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/df4e23d1-4e46-4a97-a281-482cdab78f81_isrForCrosstalkSources_1125894_58.stderr parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3013c838a340863a700b3efe258d42d9 with result from task 542 parsl.dataflow.dflow DEBUG: Task 542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/e7f47c40-4932-439d-b5e8-b2308e1a676e_isrForCrosstalkSources_1125943_62.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/e7f47c40-4932-439d-b5e8-b2308e1a676e_isrForCrosstalkSources_1125943_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:25 proc_lsst.multi:146 [INFO] found job 16896572 in provider checkpoint proc_lsst.multi INFO: found job 16896572 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:26 proc_lsst.multi:146 [INFO] found job 16896573 in provider checkpoint proc_lsst.multi INFO: found job 16896573 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e01dd06a2f3d52db9cc4c7b97818aa00 with result from task 559 parsl.dataflow.dflow DEBUG: Task 559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/8275782f-98d1-4929-9c2a-d53996efdb4f_isrForCrosstalkSources_1125946_39.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/8275782f-98d1-4929-9c2a-d53996efdb4f_isrForCrosstalkSources_1125946_39.stderr parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 126625a47eec58d9d0425fdc91d0e556 with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/1ea81f94-db4c-4010-8d7b-3200e5fbe41e_isrForCrosstalkSources_1125937_31.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/1ea81f94-db4c-4010-8d7b-3200e5fbe41e_isrForCrosstalkSources_1125937_31.stderr parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04442d79422d0d24aa0d251fea2ae2d4 with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/0242544f-785c-4882-9dde-1c15edfe8094_isrForCrosstalkSources_1125912_15.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/0242544f-785c-4882-9dde-1c15edfe8094_isrForCrosstalkSources_1125912_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:26 proc_lsst.multi:146 [INFO] found job 16896574 in provider checkpoint proc_lsst.multi INFO: found job 16896574 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd7fdbd7ea7f73f40d3e92fb18c33072 with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/d3d28935-b3ad-45f2-8fa9-779473e974d7_isrForCrosstalkSources_1125917_33.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/d3d28935-b3ad-45f2-8fa9-779473e974d7_isrForCrosstalkSources_1125917_33.stderr parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e568eeaa0166268f98e04298425b326 with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/b99f1c33-4bda-4f98-a8b9-5115b60bc799_isrForCrosstalkSources_1125912_4.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/b99f1c33-4bda-4f98-a8b9-5115b60bc799_isrForCrosstalkSources_1125912_4.stderr parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7f6741ebd615dbb5dcd6e6b0a03dc79 with result from task 530 parsl.dataflow.dflow DEBUG: Task 530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/51a36469-e70e-4d4f-a04b-43cf629079e9_isrForCrosstalkSources_1125903_34.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/51a36469-e70e-4d4f-a04b-43cf629079e9_isrForCrosstalkSources_1125903_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:26 proc_lsst.multi:146 [INFO] found job 16896575 in provider checkpoint proc_lsst.multi INFO: found job 16896575 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81563a08628975dc4d91bae374584fd0 with result from task 560 parsl.dataflow.dflow DEBUG: Task 560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/e91eb781-cbe0-45ca-a8de-3e103f40362e_isrForCrosstalkSources_1125936_26.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/e91eb781-cbe0-45ca-a8de-3e103f40362e_isrForCrosstalkSources_1125936_26.stderr parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9678ed4da6401806ef12f6d9ec139b96 with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/6d59bb72-d1bf-4b86-ab0b-93ac2e732856_isrForCrosstalkSources_1125898_14.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/6d59bb72-d1bf-4b86-ab0b-93ac2e732856_isrForCrosstalkSources_1125898_14.stderr parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 537c82bf351774f82200d4b4f0c784c9 with result from task 529 parsl.dataflow.dflow DEBUG: Task 529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0f4bf9f8-8d71-4f8e-b094-09bb761627fa_isrForCrosstalkSources_1125914_4.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0f4bf9f8-8d71-4f8e-b094-09bb761627fa_isrForCrosstalkSources_1125914_4.stderr parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86f83da29e14fe476b0d5968055ecf70 with result from task 561 parsl.dataflow.dflow DEBUG: Task 561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/26335c2e-bcf5-459a-8009-9c4ee28857e1_isrForCrosstalkSources_1125913_21.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/26335c2e-bcf5-459a-8009-9c4ee28857e1_isrForCrosstalkSources_1125913_21.stderr parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b205af48110cf9466b545d62cb565019 with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/0bec6ca9-3bf4-43d1-aebc-cda1e6488614_isrForCrosstalkSources_1125944_24.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/0bec6ca9-3bf4-43d1-aebc-cda1e6488614_isrForCrosstalkSources_1125944_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:27 proc_lsst.multi:146 [INFO] found job 16896576 in provider checkpoint proc_lsst.multi INFO: found job 16896576 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6650dc3c06c068f2caf22b57c30d259 with result from task 562 parsl.dataflow.dflow DEBUG: Task 562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/3205d968-6d13-47bc-993d-128bd46fa6ce_isrForCrosstalkSources_1125938_33.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/3205d968-6d13-47bc-993d-128bd46fa6ce_isrForCrosstalkSources_1125938_33.stderr parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41c03da7c8aa2e1cf7b604df0b7f600e with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/b1e4f637-d440-4115-ac99-100f1b993bd7_isrForCrosstalkSources_1125905_59.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/b1e4f637-d440-4115-ac99-100f1b993bd7_isrForCrosstalkSources_1125905_59.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4120b9497e076dff06ed4303965ff421 with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/d1eeaedf-9443-4616-88c2-d533b73753a4_isrForCrosstalkSources_1125926_11.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/d1eeaedf-9443-4616-88c2-d533b73753a4_isrForCrosstalkSources_1125926_11.stderr parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a150ee420936cbb95aa99cb0ae62fc0 with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/67ccf0fe-54e0-4eb6-958a-a28f7badaee0_isrForCrosstalkSources_1125942_30.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/67ccf0fe-54e0-4eb6-958a-a28f7badaee0_isrForCrosstalkSources_1125942_30.stderr parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3062b51b1d417dcc60180a9c58f96d8 with result from task 564 parsl.dataflow.dflow DEBUG: Task 564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/4116fa4f-ea41-4d14-82ed-cde4dba5bc10_isrForCrosstalkSources_1125902_12.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/4116fa4f-ea41-4d14-82ed-cde4dba5bc10_isrForCrosstalkSources_1125902_12.stderr parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35526b711b559415570f6d8bb7e382dd with result from task 558 parsl.dataflow.dflow DEBUG: Task 558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/fc35e5f6-d8c8-44dd-b394-2cfa1a837d88_isrForCrosstalkSources_1125910_55.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/fc35e5f6-d8c8-44dd-b394-2cfa1a837d88_isrForCrosstalkSources_1125910_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:27 proc_lsst.multi:146 [INFO] found job 16896577 in provider checkpoint proc_lsst.multi INFO: found job 16896577 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d248d93eb4cda3ff7925cf07dda06567 with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/4b61d1b7-a5cc-4027-b0bd-7ce58a718354_isrForCrosstalkSources_1125903_3.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/4b61d1b7-a5cc-4027-b0bd-7ce58a718354_isrForCrosstalkSources_1125903_3.stderr parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ae8bd126e2aa1337b2db22033e48440 with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/7bbe2948-7dac-4fbc-adca-5414405da6a8_isrForCrosstalkSources_1125906_28.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/7bbe2948-7dac-4fbc-adca-5414405da6a8_isrForCrosstalkSources_1125906_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:28 proc_lsst.multi:146 [INFO] found job 16896578 in provider checkpoint proc_lsst.multi INFO: found job 16896578 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31409ad08236ed6f4a2ece67896e7cb3 with result from task 535 parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/f393d083-33f3-4b74-b3ed-0b617db0a8aa_isrForCrosstalkSources_1125936_15.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/f393d083-33f3-4b74-b3ed-0b617db0a8aa_isrForCrosstalkSources_1125936_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:28 proc_lsst.multi:146 [INFO] found job 16896579 in provider checkpoint proc_lsst.multi INFO: found job 16896579 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3eac9ece03e59e95f2aabda73aa6fb29 with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/403f0e41-e520-4062-8af9-580ea7b9c180_isrForCrosstalkSources_1125918_53.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/403f0e41-e520-4062-8af9-580ea7b9c180_isrForCrosstalkSources_1125918_53.stderr parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ceeb9e3a33c257452dd7a6e64ff3ee2 with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/bde58214-4041-403f-b875-1320d69f6259_isrForCrosstalkSources_1125897_54.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/bde58214-4041-403f-b875-1320d69f6259_isrForCrosstalkSources_1125897_54.stderr parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0389ca8866da525523c43e8af2454ab with result from task 568 parsl.dataflow.dflow DEBUG: Task 568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/04bfaa9e-7cf2-463a-95be-0257d1af52f2_isrForCrosstalkSources_1125934_2.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/04bfaa9e-7cf2-463a-95be-0257d1af52f2_isrForCrosstalkSources_1125934_2.stderr parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1eaa87eb37dbc9ed0171b6f1269881e2 with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/4ed29e5e-b9b8-41b4-bda6-f462e8a6d6e0_isrForCrosstalkSources_1125900_5.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/4ed29e5e-b9b8-41b4-bda6-f462e8a6d6e0_isrForCrosstalkSources_1125900_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:29 proc_lsst.multi:146 [INFO] found job 16896580 in provider checkpoint proc_lsst.multi INFO: found job 16896580 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11be9ae849395291cd6a65ce3e0b1f38 with result from task 541 parsl.dataflow.dflow DEBUG: Task 541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/2108a32b-7646-4b2a-ab5f-17c1df068b36_isrForCrosstalkSources_1125913_31.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/2108a32b-7646-4b2a-ab5f-17c1df068b36_isrForCrosstalkSources_1125913_31.stderr parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f631811fdb19d4f3d4e281eaa6ce743f with result from task 556 parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/23cd4be0-c3b5-4bf9-9860-17225dd604dc_isrForCrosstalkSources_1125939_14.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/23cd4be0-c3b5-4bf9-9860-17225dd604dc_isrForCrosstalkSources_1125939_14.stderr parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56da911a43605c5b2b3e54a5559f427f with result from task 565 parsl.dataflow.dflow DEBUG: Task 565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/122a9042-cb03-4a7c-8b42-14fba991d0dd_isrForCrosstalkSources_1125918_17.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/122a9042-cb03-4a7c-8b42-14fba991d0dd_isrForCrosstalkSources_1125918_17.stderr parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57bfe241de1d9687cc55ce6e996fb375 with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/9828268b-4d7b-45e7-8dc9-d4f40a37754b_isrForCrosstalkSources_1125895_8.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/9828268b-4d7b-45e7-8dc9-d4f40a37754b_isrForCrosstalkSources_1125895_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:29 proc_lsst.multi:146 [INFO] found job 16896581 in provider checkpoint proc_lsst.multi INFO: found job 16896581 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d8b3c745b4a6d33eedd9ec3d46dd53f with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a9501c5a-b263-4b44-bb71-12df24b98a59_isrForCrosstalkSources_1125937_61.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a9501c5a-b263-4b44-bb71-12df24b98a59_isrForCrosstalkSources_1125937_61.stderr parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e3974f6fb04ab2af9cbdfcd8665cee4 with result from task 539 parsl.dataflow.dflow DEBUG: Task 539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/a25dda5c-fbaf-4bc7-81b5-cd2d2b74262c_isrForCrosstalkSources_1125926_58.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/a25dda5c-fbaf-4bc7-81b5-cd2d2b74262c_isrForCrosstalkSources_1125926_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:29 proc_lsst.multi:146 [INFO] found job 16896582 in provider checkpoint proc_lsst.multi INFO: found job 16896582 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03cd6a929fd4525a1c39d462969d3574 with result from task 567 parsl.dataflow.dflow DEBUG: Task 567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/70e361a6-3450-4925-aa43-bc9a388fbb7e_isrForCrosstalkSources_1125907_9.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/70e361a6-3450-4925-aa43-bc9a388fbb7e_isrForCrosstalkSources_1125907_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:30 proc_lsst.multi:146 [INFO] found job 16896583 in provider checkpoint proc_lsst.multi INFO: found job 16896583 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a01dd96f6c30d4d24305c3a5d1a0190d with result from task 563 parsl.dataflow.dflow DEBUG: Task 563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/73b658b6-fdd2-4390-82ee-650ba81f681f_isrForCrosstalkSources_1125938_3.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/73b658b6-fdd2-4390-82ee-650ba81f681f_isrForCrosstalkSources_1125938_3.stderr parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28e3ca1fd4dcd571ee66601a5b868c0d with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/83298bbb-22f3-4938-af7e-46a687d77acc_isrForCrosstalkSources_1125925_60.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/83298bbb-22f3-4938-af7e-46a687d77acc_isrForCrosstalkSources_1125925_60.stderr parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d37f88c9b8cf1ead1164a2425808884 with result from task 566 parsl.dataflow.dflow DEBUG: Task 566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/fa9eb24d-22a7-4018-87df-1cc3e16283c4_isrForCrosstalkSources_1125913_52.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/fa9eb24d-22a7-4018-87df-1cc3e16283c4_isrForCrosstalkSources_1125913_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:31 proc_lsst.multi:146 [INFO] found job 16896584 in provider checkpoint proc_lsst.multi INFO: found job 16896584 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7980cfdf7bf0c8eb0b7117e587fdc357 with result from task 577 parsl.dataflow.dflow DEBUG: Task 577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/4c10aaa9-cec2-4e10-b06f-37c9a67a9447_isrForCrosstalkSources_1125924_40.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/4c10aaa9-cec2-4e10-b06f-37c9a67a9447_isrForCrosstalkSources_1125924_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:31 proc_lsst.multi:146 [INFO] found job 16896585 in provider checkpoint proc_lsst.multi INFO: found job 16896585 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:32 proc_lsst.multi:146 [INFO] found job 16896586 in provider checkpoint proc_lsst.multi INFO: found job 16896586 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:32 proc_lsst.multi:146 [INFO] found job 16896587 in provider checkpoint proc_lsst.multi INFO: found job 16896587 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:32 proc_lsst.multi:146 [INFO] found job 16896588 in provider checkpoint proc_lsst.multi INFO: found job 16896588 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:32 proc_lsst.multi:146 [INFO] found job 16896589 in provider checkpoint proc_lsst.multi INFO: found job 16896589 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:33 proc_lsst.multi:146 [INFO] found job 16896590 in provider checkpoint proc_lsst.multi INFO: found job 16896590 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:33 proc_lsst.multi:146 [INFO] found job 16896591 in provider checkpoint proc_lsst.multi INFO: found job 16896591 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:34 proc_lsst.multi:146 [INFO] found job 16896592 in provider checkpoint proc_lsst.multi INFO: found job 16896592 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d115167f04a999253fd03b796814609 with result from task 586 parsl.dataflow.dflow DEBUG: Task 586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/216a98d9-2aeb-43c4-9847-4f46159326c1_isrForCrosstalkSources_1125935_11.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/216a98d9-2aeb-43c4-9847-4f46159326c1_isrForCrosstalkSources_1125935_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:34 proc_lsst.multi:146 [INFO] found job 16896593 in provider checkpoint proc_lsst.multi INFO: found job 16896593 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:35 proc_lsst.multi:146 [INFO] found job 16896594 in provider checkpoint proc_lsst.multi INFO: found job 16896594 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:35 proc_lsst.multi:146 [INFO] found job 16896595 in provider checkpoint proc_lsst.multi INFO: found job 16896595 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84032aba461701aacbf360c7d74ec255 with result from task 587 parsl.dataflow.dflow DEBUG: Task 587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/0918d60e-0805-45de-8219-04451dd735f9_isrForCrosstalkSources_1125915_10.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/0918d60e-0805-45de-8219-04451dd735f9_isrForCrosstalkSources_1125915_10.stderr parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2701146bf0193ad2481e8804f5cf387a with result from task 598 parsl.dataflow.dflow DEBUG: Task 598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/c09e6a9d-0bac-475b-b673-96d16db269f4_isrForCrosstalkSources_1125915_22.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/c09e6a9d-0bac-475b-b673-96d16db269f4_isrForCrosstalkSources_1125915_22.stderr parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a491971e5ea7de76fbc8ff2532dafeb7 with result from task 601 parsl.dataflow.dflow DEBUG: Task 601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/47bcc5fa-dbed-48fe-966f-21e1afd81d02_isrForCrosstalkSources_1125939_1.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/47bcc5fa-dbed-48fe-966f-21e1afd81d02_isrForCrosstalkSources_1125939_1.stderr parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 552a8ed995fe255f720c638fa709ff2d with result from task 595 parsl.dataflow.dflow DEBUG: Task 595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/4ba1c79a-02a1-4702-be3f-602ecd374cd5_isrForCrosstalkSources_1125939_17.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/4ba1c79a-02a1-4702-be3f-602ecd374cd5_isrForCrosstalkSources_1125939_17.stderr parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c9b0444d5972a4cd02a69b0aca143a9 with result from task 592 parsl.dataflow.dflow DEBUG: Task 592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/f14f624a-ff79-4b8a-acb4-de04f2008869_isrForCrosstalkSources_1125927_30.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/f14f624a-ff79-4b8a-acb4-de04f2008869_isrForCrosstalkSources_1125927_30.stderr parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e8dbe11ac4cb0191fdfe5031dbe94ef with result from task 588 parsl.dataflow.dflow DEBUG: Task 588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/756883fc-9048-48dd-b10a-0cb0fc7c7de7_isrForCrosstalkSources_1125892_37.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/756883fc-9048-48dd-b10a-0cb0fc7c7de7_isrForCrosstalkSources_1125892_37.stderr parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed425979e65923d444d915cf0b6c8ff5 with result from task 597 parsl.dataflow.dflow DEBUG: Task 597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/de8799d8-20d3-4c9d-b247-3509bde0dfd8_isrForCrosstalkSources_1125909_32.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/de8799d8-20d3-4c9d-b247-3509bde0dfd8_isrForCrosstalkSources_1125909_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:36 proc_lsst.multi:146 [INFO] found job 16896596 in provider checkpoint proc_lsst.multi INFO: found job 16896596 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:36 proc_lsst.multi:146 [INFO] found job 16896597 in provider checkpoint proc_lsst.multi INFO: found job 16896597 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b4f4df511bfbfc132947f643b1b7894 with result from task 608 parsl.dataflow.dflow DEBUG: Task 608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/a8e08cc1-fae7-4360-95f3-9e0d543109f4_isrForCrosstalkSources_1125929_49.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/a8e08cc1-fae7-4360-95f3-9e0d543109f4_isrForCrosstalkSources_1125929_49.stderr parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e6cd41aaefbf8ef2662024907df238d with result from task 602 parsl.dataflow.dflow DEBUG: Task 602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/74fd87d4-1e4c-446b-939e-4a77c118a192_isrForCrosstalkSources_1125919_15.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/74fd87d4-1e4c-446b-939e-4a77c118a192_isrForCrosstalkSources_1125919_15.stderr parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 134406582bb4c2a172d6f098ccb1d1cc with result from task 600 parsl.dataflow.dflow DEBUG: Task 600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/b82d477a-20fc-40b7-9747-946a3dbfbf1b_isrForCrosstalkSources_1125911_3.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/b82d477a-20fc-40b7-9747-946a3dbfbf1b_isrForCrosstalkSources_1125911_3.stderr parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d43ffbbcc50a001fdaf097ebc19227e5 with result from task 604 parsl.dataflow.dflow DEBUG: Task 604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/eaf18f41-8662-444f-8ac5-b4fe23fac25a_isrForCrosstalkSources_1125906_48.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/eaf18f41-8662-444f-8ac5-b4fe23fac25a_isrForCrosstalkSources_1125906_48.stderr parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de8a6b0ff03dd1f886dc67f31a831d10 with result from task 596 parsl.dataflow.dflow DEBUG: Task 596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/278cf566-947b-4ae1-a728-4564a996cde7_isrForCrosstalkSources_1125929_24.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/278cf566-947b-4ae1-a728-4564a996cde7_isrForCrosstalkSources_1125929_24.stderr parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5a59d4af0d0dd7bd35dc47e9d9e0f5b with result from task 609 parsl.dataflow.dflow DEBUG: Task 609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/cbf6137e-b302-4f5d-968c-067e4e774a04_isrForCrosstalkSources_1125899_26.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/cbf6137e-b302-4f5d-968c-067e4e774a04_isrForCrosstalkSources_1125899_26.stderr parsl.dataflow.dflow INFO: Task 611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 647057a4f31bd1a21daa7e80d2fe9e43 with result from task 611 parsl.dataflow.dflow DEBUG: Task 611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/28e44312-74f3-4a1e-9118-2e01f2613427_isrForCrosstalkSources_1125913_18.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/28e44312-74f3-4a1e-9118-2e01f2613427_isrForCrosstalkSources_1125913_18.stderr parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5570ffbaa6220232362fa204bce9ac6d with result from task 615 parsl.dataflow.dflow DEBUG: Task 615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/b3fabe6c-5fcd-4bb8-92b5-7d8d3f95a732_isrForCrosstalkSources_1125907_8.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/b3fabe6c-5fcd-4bb8-92b5-7d8d3f95a732_isrForCrosstalkSources_1125907_8.stderr parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a0ad7095f23df1f36c664f2a3a5f9a0 with result from task 578 parsl.dataflow.dflow DEBUG: Task 578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/489eb157-43ff-4e54-be3a-6f6b1244b22c_isrForCrosstalkSources_1125899_40.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/489eb157-43ff-4e54-be3a-6f6b1244b22c_isrForCrosstalkSources_1125899_40.stderr parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8eb37052b683d61b567d2d3c7a755745 with result from task 582 parsl.dataflow.dflow DEBUG: Task 582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/114b86a5-12ec-4834-a55c-b926f848e340_isrForCrosstalkSources_1125899_6.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/114b86a5-12ec-4834-a55c-b926f848e340_isrForCrosstalkSources_1125899_6.stderr parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18a7a8ae4323e1d5265a19fd709ef321 with result from task 613 parsl.dataflow.dflow DEBUG: Task 613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/632e2e2e-ab8b-4659-9b15-d5ea5c74f23e_isrForCrosstalkSources_1125934_27.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/632e2e2e-ab8b-4659-9b15-d5ea5c74f23e_isrForCrosstalkSources_1125934_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:39 proc_lsst.multi:146 [INFO] found job 16896598 in provider checkpoint proc_lsst.multi INFO: found job 16896598 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 977b3f177d6f83c767bf0a3d8924dc88 with result from task 593 parsl.dataflow.dflow DEBUG: Task 593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/ad1e230d-784c-421f-9c2d-00ba3e89f3d3_isrForCrosstalkSources_1125893_61.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/ad1e230d-784c-421f-9c2d-00ba3e89f3d3_isrForCrosstalkSources_1125893_61.stderr parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25e1fe53352205a2a6f31e9cd1769a42 with result from task 580 parsl.dataflow.dflow DEBUG: Task 580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/c6d73bad-d701-4698-a776-5e78715f6ff0_isrForCrosstalkSources_1125899_38.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/c6d73bad-d701-4698-a776-5e78715f6ff0_isrForCrosstalkSources_1125899_38.stderr parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a99189230d852a83c6932ed14da461f with result from task 581 parsl.dataflow.dflow DEBUG: Task 581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a25cb8e0-9a1c-48a4-b4e2-1ed482f1bf98_isrForCrosstalkSources_1125921_7.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a25cb8e0-9a1c-48a4-b4e2-1ed482f1bf98_isrForCrosstalkSources_1125921_7.stderr parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 789bbe7d272b00a5c518990e539bf542 with result from task 591 parsl.dataflow.dflow DEBUG: Task 591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ff361bb0-236a-479e-89ee-6b442ad1c7c1_isrForCrosstalkSources_1125918_33.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ff361bb0-236a-479e-89ee-6b442ad1c7c1_isrForCrosstalkSources_1125918_33.stderr parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 309c695f6119cc7fa5b220b7798e3dd4 with result from task 594 parsl.dataflow.dflow DEBUG: Task 594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/c6f70afe-a821-4712-afdf-4a4aa8260c34_isrForCrosstalkSources_1125912_46.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/c6f70afe-a821-4712-afdf-4a4aa8260c34_isrForCrosstalkSources_1125912_46.stderr parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cb6f7718f5fb362292037035b5c8e1b with result from task 607 parsl.dataflow.dflow DEBUG: Task 607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/9eefe7f1-bcff-45d9-b24c-7dab775af627_isrForCrosstalkSources_1125931_40.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/9eefe7f1-bcff-45d9-b24c-7dab775af627_isrForCrosstalkSources_1125931_40.stderr parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c1cb7924f13b7748aed94a19ab4d79e with result from task 599 parsl.dataflow.dflow DEBUG: Task 599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/ee658c11-0c8b-4f2c-b39f-4c55b780ab64_isrForCrosstalkSources_1125891_45.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/ee658c11-0c8b-4f2c-b39f-4c55b780ab64_isrForCrosstalkSources_1125891_45.stderr parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5630ce6479a2078f97f3d1cc28bc7f4b with result from task 585 parsl.dataflow.dflow DEBUG: Task 585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/ffffd5dc-92c1-41d9-91f2-1cffe42235de_isrForCrosstalkSources_1125892_35.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/ffffd5dc-92c1-41d9-91f2-1cffe42235de_isrForCrosstalkSources_1125892_35.stderr parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cc05cbd09aacf8d3161c5b132fcff33 with result from task 579 parsl.dataflow.dflow DEBUG: Task 579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/8d528bb9-3294-4825-babf-76abdadb0394_isrForCrosstalkSources_1125907_2.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/8d528bb9-3294-4825-babf-76abdadb0394_isrForCrosstalkSources_1125907_2.stderr parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bfa2e587a5bba958301ff2cb9cc8fbe with result from task 583 parsl.dataflow.dflow DEBUG: Task 583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/b4e174b6-bd11-4132-a62b-edd0dfb47b28_isrForCrosstalkSources_1125936_49.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/b4e174b6-bd11-4132-a62b-edd0dfb47b28_isrForCrosstalkSources_1125936_49.stderr parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5c8b7425c1fdbe886f7b98cd582f062 with result from task 605 parsl.dataflow.dflow DEBUG: Task 605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/352c8190-88f6-444c-ae21-fa25d0b1029f_isrForCrosstalkSources_1125892_52.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/352c8190-88f6-444c-ae21-fa25d0b1029f_isrForCrosstalkSources_1125892_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:39 proc_lsst.multi:146 [INFO] found job 16896599 in provider checkpoint proc_lsst.multi INFO: found job 16896599 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7da0abc2b3e5ff6e1597d42c44d53cf1 with result from task 584 parsl.dataflow.dflow DEBUG: Task 584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/bda2ceb9-b533-4547-80b4-449291e15a7f_isrForCrosstalkSources_1125916_58.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/bda2ceb9-b533-4547-80b4-449291e15a7f_isrForCrosstalkSources_1125916_58.stderr parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24acf727b16fae5dca6234d836b787b8 with result from task 618 parsl.dataflow.dflow DEBUG: Task 618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/cd5c9a3f-d613-426a-bc99-9cbab60b31a3_isrForCrosstalkSources_1125933_18.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/cd5c9a3f-d613-426a-bc99-9cbab60b31a3_isrForCrosstalkSources_1125933_18.stderr parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a20e42bdf8d8e3207fa445b0a119ef16 with result from task 589 parsl.dataflow.dflow DEBUG: Task 589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/4e6ee0ab-41f7-476a-ac5d-21175ee48989_isrForCrosstalkSources_1125909_12.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/4e6ee0ab-41f7-476a-ac5d-21175ee48989_isrForCrosstalkSources_1125909_12.stderr parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f26fa6dcd5d87c585aa956545287955 with result from task 590 parsl.dataflow.dflow DEBUG: Task 590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/4077a22f-d85d-422f-bfac-495dabfab365_isrForCrosstalkSources_1125946_45.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/4077a22f-d85d-422f-bfac-495dabfab365_isrForCrosstalkSources_1125946_45.stderr parsl.dataflow.dflow INFO: Task 610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73bf4088ea03e47d82a479bc64190213 with result from task 610 parsl.dataflow.dflow DEBUG: Task 610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/7b46b0ba-d9d2-4fdb-b6a4-296c73c92627_isrForCrosstalkSources_1125914_38.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/7b46b0ba-d9d2-4fdb-b6a4-296c73c92627_isrForCrosstalkSources_1125914_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:40 proc_lsst.multi:146 [INFO] found job 16896600 in provider checkpoint proc_lsst.multi INFO: found job 16896600 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84b3259f5fa1416d1ef3c45adb8ff00a with result from task 603 parsl.dataflow.dflow DEBUG: Task 603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/b85ccc6c-cb02-49df-b0fb-51f03c89d5e4_isrForCrosstalkSources_1125888_21.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/b85ccc6c-cb02-49df-b0fb-51f03c89d5e4_isrForCrosstalkSources_1125888_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:41 proc_lsst.multi:146 [INFO] found job 16896601 in provider checkpoint proc_lsst.multi INFO: found job 16896601 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d218166444a99a26fafbff9d5dcbc00d with result from task 620 parsl.dataflow.dflow DEBUG: Task 620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/2dd13774-696b-4227-bc62-1a8c9ec833e6_isrForCrosstalkSources_1125907_16.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/2dd13774-696b-4227-bc62-1a8c9ec833e6_isrForCrosstalkSources_1125907_16.stderr parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 037a6b251cc49d75d3e3d0c1f962179d with result from task 619 parsl.dataflow.dflow DEBUG: Task 619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/14088ea3-a470-4287-86a3-f02987ae70ed_isrForCrosstalkSources_1125942_55.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/14088ea3-a470-4287-86a3-f02987ae70ed_isrForCrosstalkSources_1125942_55.stderr parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e2c9638891f75ebad0ae5b9f96aac19 with result from task 616 parsl.dataflow.dflow DEBUG: Task 616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d4ecde88-4633-469b-9182-a6ea7857b013_isrForCrosstalkSources_1125905_55.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d4ecde88-4633-469b-9182-a6ea7857b013_isrForCrosstalkSources_1125905_55.stderr parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ab03d67355b22925c6a2174e7a40fde with result from task 617 parsl.dataflow.dflow DEBUG: Task 617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f64ac83b-9730-454d-9239-92dca3bf3a81_isrForCrosstalkSources_1125910_54.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f64ac83b-9730-454d-9239-92dca3bf3a81_isrForCrosstalkSources_1125910_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:41 proc_lsst.multi:146 [INFO] found job 16896602 in provider checkpoint proc_lsst.multi INFO: found job 16896602 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d771b4383d8696b8363a111ba0000e7 with result from task 606 parsl.dataflow.dflow DEBUG: Task 606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/903bc0b1-ae99-42dc-a5fb-ae58c4eace0d_isrForCrosstalkSources_1125943_53.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/903bc0b1-ae99-42dc-a5fb-ae58c4eace0d_isrForCrosstalkSources_1125943_53.stderr parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 623eb2aff9c4b52373411d5cb9417f72 with result from task 612 parsl.dataflow.dflow DEBUG: Task 612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/7e92762a-1011-4917-a988-c9497a94fc49_isrForCrosstalkSources_1125926_62.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/7e92762a-1011-4917-a988-c9497a94fc49_isrForCrosstalkSources_1125926_62.stderr parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ede0c0c7f7b7164f149cdc05eb92fb7 with result from task 614 parsl.dataflow.dflow DEBUG: Task 614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/c2c890a5-5c62-4678-acc5-445c6cbc1132_isrForCrosstalkSources_1125932_12.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/c2c890a5-5c62-4678-acc5-445c6cbc1132_isrForCrosstalkSources_1125932_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:42 proc_lsst.multi:146 [INFO] found job 16896603 in provider checkpoint proc_lsst.multi INFO: found job 16896603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:42 proc_lsst.multi:146 [INFO] found job 16896604 in provider checkpoint proc_lsst.multi INFO: found job 16896604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:42 proc_lsst.multi:146 [INFO] found job 16896605 in provider checkpoint proc_lsst.multi INFO: found job 16896605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6838217aed4df9582fc6697d36fc08bf with result from task 621 parsl.dataflow.dflow DEBUG: Task 621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/9b887f82-9ca0-4ac8-a616-d6e476fd2ec9_isrForCrosstalkSources_1125929_46.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/9b887f82-9ca0-4ac8-a616-d6e476fd2ec9_isrForCrosstalkSources_1125929_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:43 proc_lsst.multi:146 [INFO] found job 16896606 in provider checkpoint proc_lsst.multi INFO: found job 16896606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:43 proc_lsst.multi:146 [INFO] found job 16896607 in provider checkpoint proc_lsst.multi INFO: found job 16896607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:44 proc_lsst.multi:146 [INFO] found job 16896608 in provider checkpoint proc_lsst.multi INFO: found job 16896608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03d458e8a4492c7f6666b591699eaa8c with result from task 624 parsl.dataflow.dflow DEBUG: Task 624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/fa53cbb1-44a4-45f7-aa0c-be060775b53c_isrForCrosstalkSources_1125918_5.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/fa53cbb1-44a4-45f7-aa0c-be060775b53c_isrForCrosstalkSources_1125918_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:44 proc_lsst.multi:146 [INFO] found job 16896609 in provider checkpoint proc_lsst.multi INFO: found job 16896609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2402e8a1e7fcc29264f72678a76c568e with result from task 622 parsl.dataflow.dflow DEBUG: Task 622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/e1e7a69a-99d0-4b92-adf9-663b24a3c869_isrForCrosstalkSources_1125897_25.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/e1e7a69a-99d0-4b92-adf9-663b24a3c869_isrForCrosstalkSources_1125897_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:45 proc_lsst.multi:146 [INFO] found job 16896610 in provider checkpoint proc_lsst.multi INFO: found job 16896610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b008543cc9d28a82d3838d207fd1111f with result from task 629 parsl.dataflow.dflow DEBUG: Task 629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/b2ac82ed-b5d8-4769-8b2f-80ef588289ca_isrForCrosstalkSources_1125947_2.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/b2ac82ed-b5d8-4769-8b2f-80ef588289ca_isrForCrosstalkSources_1125947_2.stderr parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb259027cc047f750e8c5f116506ef06 with result from task 625 parsl.dataflow.dflow DEBUG: Task 625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/1f4c3c81-2c75-406c-9c40-a5be296601cd_isrForCrosstalkSources_1125947_16.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/1f4c3c81-2c75-406c-9c40-a5be296601cd_isrForCrosstalkSources_1125947_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:45 proc_lsst.multi:146 [INFO] found job 16896611 in provider checkpoint proc_lsst.multi INFO: found job 16896611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 777c8bbb832366fe2d307db0604547a1 with result from task 627 parsl.dataflow.dflow DEBUG: Task 627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/60f9094c-9e2a-4d08-be09-101e8fb83cde_isrForCrosstalkSources_1125917_6.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/60f9094c-9e2a-4d08-be09-101e8fb83cde_isrForCrosstalkSources_1125917_6.stderr parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b015335a860c6d1e32e41a1d57a1e15 with result from task 626 parsl.dataflow.dflow DEBUG: Task 626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/3434b134-28d6-479b-8afb-30754c27c851_isrForCrosstalkSources_1125945_42.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/3434b134-28d6-479b-8afb-30754c27c851_isrForCrosstalkSources_1125945_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:46 proc_lsst.multi:146 [INFO] found job 16896612 in provider checkpoint proc_lsst.multi INFO: found job 16896612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca102f620197d9802c911491b09d9dbc with result from task 623 parsl.dataflow.dflow DEBUG: Task 623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/3bf1311d-6e7e-434f-82b2-87369be7eaab_isrForCrosstalkSources_1125918_9.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/3bf1311d-6e7e-434f-82b2-87369be7eaab_isrForCrosstalkSources_1125918_9.stderr parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9470ed7c5099f469d1bc1524372800ea with result from task 630 parsl.dataflow.dflow DEBUG: Task 630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/11f4da60-2540-4f5d-b1ca-790624ff4a52_isrForCrosstalkSources_1125900_60.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/11f4da60-2540-4f5d-b1ca-790624ff4a52_isrForCrosstalkSources_1125900_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:46 proc_lsst.multi:146 [INFO] found job 16896613 in provider checkpoint proc_lsst.multi INFO: found job 16896613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c77d091a1c37cd3c44cf37d92a2a134d with result from task 632 parsl.dataflow.dflow DEBUG: Task 632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/bede607b-db3f-4c78-9e6b-a59a79f2a155_isrForCrosstalkSources_1125928_32.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/bede607b-db3f-4c78-9e6b-a59a79f2a155_isrForCrosstalkSources_1125928_32.stderr parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55af819dc3f80ace60d13f9c743c6574 with result from task 631 parsl.dataflow.dflow DEBUG: Task 631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/bbe5a108-dc3b-4bc2-bfa6-ec16a39ea6ff_isrForCrosstalkSources_1125923_1.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/bbe5a108-dc3b-4bc2-bfa6-ec16a39ea6ff_isrForCrosstalkSources_1125923_1.stderr parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db10b7d00848978d32a2113c543dfca6 with result from task 628 parsl.dataflow.dflow DEBUG: Task 628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/e828da5c-843c-4eb9-b3a7-48182eb4cf37_isrForCrosstalkSources_1125893_14.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/e828da5c-843c-4eb9-b3a7-48182eb4cf37_isrForCrosstalkSources_1125893_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:47 proc_lsst.multi:146 [INFO] found job 16896614 in provider checkpoint proc_lsst.multi INFO: found job 16896614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e45bdc12c890cbb0902bffced1915e5d with result from task 633 parsl.dataflow.dflow DEBUG: Task 633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8b1f1eb1-ed39-4371-8699-bfec2a85699e_isrForCrosstalkSources_1125891_19.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8b1f1eb1-ed39-4371-8699-bfec2a85699e_isrForCrosstalkSources_1125891_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:47 proc_lsst.multi:146 [INFO] found job 16896615 in provider checkpoint proc_lsst.multi INFO: found job 16896615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:48 proc_lsst.multi:146 [INFO] found job 16896616 in provider checkpoint proc_lsst.multi INFO: found job 16896616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:48 proc_lsst.multi:146 [INFO] found job 16896617 in provider checkpoint proc_lsst.multi INFO: found job 16896617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:49 proc_lsst.multi:146 [INFO] found job 16896618 in provider checkpoint proc_lsst.multi INFO: found job 16896618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:49 proc_lsst.multi:146 [INFO] found job 16896619 in provider checkpoint proc_lsst.multi INFO: found job 16896619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22cea8a089819bd6574a71f9b5b4548c with result from task 646 parsl.dataflow.dflow DEBUG: Task 646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/75f7d147-6341-48e0-b3c9-c4acded97e3b_isrForCrosstalkSources_1125913_27.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/75f7d147-6341-48e0-b3c9-c4acded97e3b_isrForCrosstalkSources_1125913_27.stderr parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59c13ddb3eaa9b5d50728f54f140d096 with result from task 661 parsl.dataflow.dflow DEBUG: Task 661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/d3aad5ad-29e2-4182-beb4-3f89bc4a9927_isrForCrosstalkSources_1125926_25.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/d3aad5ad-29e2-4182-beb4-3f89bc4a9927_isrForCrosstalkSources_1125926_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:50 proc_lsst.multi:146 [INFO] found job 16896620 in provider checkpoint proc_lsst.multi INFO: found job 16896620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be8c4d47b79e1b7b9705284a013a69b8 with result from task 658 parsl.dataflow.dflow DEBUG: Task 658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/44d1436d-6962-4652-93e0-137245cd32c8_isrForCrosstalkSources_1125902_19.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/44d1436d-6962-4652-93e0-137245cd32c8_isrForCrosstalkSources_1125902_19.stderr parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d439a009a2d1ba4ede2137d271faed72 with result from task 639 parsl.dataflow.dflow DEBUG: Task 639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/c165cd5c-f6d5-4838-8dc0-17c610198360_isrForCrosstalkSources_1125935_34.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/c165cd5c-f6d5-4838-8dc0-17c610198360_isrForCrosstalkSources_1125935_34.stderr parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ffa062a790bf0e62855378ff6d17404 with result from task 635 parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/ef6d7c7e-959c-4934-b32d-83d2093f5b6e_isrForCrosstalkSources_1125903_4.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/ef6d7c7e-959c-4934-b32d-83d2093f5b6e_isrForCrosstalkSources_1125903_4.stderr parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00410611cf980c99745bd8dc1cc62448 with result from task 650 parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:51 proc_lsst.multi:146 [INFO] found job 16896621 in provider checkpoint proc_lsst.multi INFO: found job 16896621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/a62fe34f-5663-4b43-883e-c4e1d4dceb1d_isrForCrosstalkSources_1125931_8.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/a62fe34f-5663-4b43-883e-c4e1d4dceb1d_isrForCrosstalkSources_1125931_8.stderr parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b5d878096be3128eec9f985969b6ffc with result from task 664 parsl.dataflow.dflow DEBUG: Task 664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/8b981db2-4805-40f6-8e13-3cbcd31ffda1_isrForCrosstalkSources_1125947_17.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/8b981db2-4805-40f6-8e13-3cbcd31ffda1_isrForCrosstalkSources_1125947_17.stderr parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71f2f5f2daf8ac65b4f6e67941e83b07 with result from task 636 parsl.dataflow.dflow DEBUG: Task 636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/9a98830c-e8c5-4c35-83eb-444eb1e1837e_isrForCrosstalkSources_1125928_45.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/9a98830c-e8c5-4c35-83eb-444eb1e1837e_isrForCrosstalkSources_1125928_45.stderr parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad32a785ed084cee19526bfe24a9e317 with result from task 642 parsl.dataflow.dflow DEBUG: Task 642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/5e20b04a-2c4c-4df3-be2b-66c38cb8759f_isrForCrosstalkSources_1125921_22.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/5e20b04a-2c4c-4df3-be2b-66c38cb8759f_isrForCrosstalkSources_1125921_22.stderr parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 017063ad028807d74f02a21db93f57cb with result from task 659 parsl.dataflow.dflow DEBUG: Task 659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/1348edcd-ae70-4c4a-8be5-be85895a8316_isrForCrosstalkSources_1125893_8.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/1348edcd-ae70-4c4a-8be5-be85895a8316_isrForCrosstalkSources_1125893_8.stderr parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd37ca56cc7586c2edf4b96a0293d00b with result from task 652 parsl.dataflow.dflow DEBUG: Task 652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/c459f0f8-7609-4871-85cf-138fc827c1a9_isrForCrosstalkSources_1125897_24.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/c459f0f8-7609-4871-85cf-138fc827c1a9_isrForCrosstalkSources_1125897_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:51 proc_lsst.multi:146 [INFO] found job 16896622 in provider checkpoint proc_lsst.multi INFO: found job 16896622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eca4db9e78662c10e536e1802354c9e6 with result from task 640 parsl.dataflow.dflow DEBUG: Task 640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/5224934c-04a6-49ec-8914-28bdb90475dc_isrForCrosstalkSources_1125902_32.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/5224934c-04a6-49ec-8914-28bdb90475dc_isrForCrosstalkSources_1125902_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:51 proc_lsst.multi:146 [INFO] found job 16896623 in provider checkpoint proc_lsst.multi INFO: found job 16896623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d916a5b01857bd8277f573050f131998 with result from task 644 parsl.dataflow.dflow DEBUG: Task 644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/dbc9d125-d061-4418-908b-297abe954460_isrForCrosstalkSources_1125935_5.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/dbc9d125-d061-4418-908b-297abe954460_isrForCrosstalkSources_1125935_5.stderr parsl.dataflow.dflow INFO: Task 656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3865e0190d76f026d86a9751e4194254 with result from task 656 parsl.dataflow.dflow DEBUG: Task 656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/e145070b-e4af-47b2-a6ff-2e268d20279d_isrForCrosstalkSources_1125918_29.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/e145070b-e4af-47b2-a6ff-2e268d20279d_isrForCrosstalkSources_1125918_29.stderr parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af35641ea225eed7797ca2daff4a1826 with result from task 662 parsl.dataflow.dflow DEBUG: Task 662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/900d159c-e297-4c0c-adbb-24ba9c1a84e9_isrForCrosstalkSources_1125893_7.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/900d159c-e297-4c0c-adbb-24ba9c1a84e9_isrForCrosstalkSources_1125893_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:52 proc_lsst.multi:146 [INFO] found job 16896624 in provider checkpoint proc_lsst.multi INFO: found job 16896624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9062c7df05a01f80d8278d96302a56d3 with result from task 569 parsl.dataflow.dflow DEBUG: Task 569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/db19c79d-d9a0-44ee-9b5e-a0e629bfbd20_isrForCrosstalkSources_1125894_19.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/db19c79d-d9a0-44ee-9b5e-a0e629bfbd20_isrForCrosstalkSources_1125894_19.stderr parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55870cf73f058997cc6fce9429bb1a03 with result from task 571 parsl.dataflow.dflow DEBUG: Task 571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/6529e91e-cee5-421c-bf43-8ce6946693d5_isrForCrosstalkSources_1125934_8.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/6529e91e-cee5-421c-bf43-8ce6946693d5_isrForCrosstalkSources_1125934_8.stderr parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93d5700941a06d195f4194c96908137f with result from task 576 parsl.dataflow.dflow DEBUG: Task 576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/b4f5d44a-cbca-4184-bbf2-fd8f7daf988b_isrForCrosstalkSources_1125907_25.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/b4f5d44a-cbca-4184-bbf2-fd8f7daf988b_isrForCrosstalkSources_1125907_25.stderr parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3e7d1ead79033b6a7147c1f90fc0d96 with result from task 575 parsl.dataflow.dflow DEBUG: Task 575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e33af064-d698-41d4-b000-31da3aae9600_isrForCrosstalkSources_1125929_5.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e33af064-d698-41d4-b000-31da3aae9600_isrForCrosstalkSources_1125929_5.stderr parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cd8a12a2539571f2e298ae828d7a26c with result from task 634 parsl.dataflow.dflow DEBUG: Task 634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/6e43801b-c305-45de-bb12-a776bee4f63e_isrForCrosstalkSources_1125928_28.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/6e43801b-c305-45de-bb12-a776bee4f63e_isrForCrosstalkSources_1125928_28.stderr parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32197bf767b097afb84a92f3ed08343f with result from task 655 parsl.dataflow.dflow DEBUG: Task 655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/90ab9046-5499-4317-b54e-f6c446840c88_isrForCrosstalkSources_1125910_44.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/90ab9046-5499-4317-b54e-f6c446840c88_isrForCrosstalkSources_1125910_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:52 proc_lsst.multi:146 [INFO] found job 16896625 in provider checkpoint proc_lsst.multi INFO: found job 16896625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68e7fb959c31c5de09b96eb375215ea9 with result from task 572 parsl.dataflow.dflow DEBUG: Task 572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/2f6afd46-03cf-4637-85e6-a188b5d5730e_isrForCrosstalkSources_1125939_37.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/2f6afd46-03cf-4637-85e6-a188b5d5730e_isrForCrosstalkSources_1125939_37.stderr parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1173a15134fc7ca09d4a6bedd8f93bc9 with result from task 574 parsl.dataflow.dflow DEBUG: Task 574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/de8a0887-45df-4dc0-a500-facf0fa5b69e_isrForCrosstalkSources_1125893_53.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/de8a0887-45df-4dc0-a500-facf0fa5b69e_isrForCrosstalkSources_1125893_53.stderr parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ea2a5ceead89834cf73470a36686307 with result from task 654 parsl.dataflow.dflow DEBUG: Task 654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/f2e24ea0-5899-4c78-abb7-aa12d49f2167_isrForCrosstalkSources_1125938_44.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/f2e24ea0-5899-4c78-abb7-aa12d49f2167_isrForCrosstalkSources_1125938_44.stderr parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e52eec7b5c3ac8dca9a8efd3c6bd7ba with result from task 573 parsl.dataflow.dflow DEBUG: Task 573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/0641f819-aff0-46c9-b01b-8bac859cce8a_isrForCrosstalkSources_1125921_44.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/0641f819-aff0-46c9-b01b-8bac859cce8a_isrForCrosstalkSources_1125921_44.stderr parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5665ec6b4186aa8bd1b1df3871ffea9 with result from task 570 parsl.dataflow.dflow DEBUG: Task 570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/56a92b19-f7ee-4cb4-a796-f81531d97f19_isrForCrosstalkSources_1125895_58.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/56a92b19-f7ee-4cb4-a796-f81531d97f19_isrForCrosstalkSources_1125895_58.stderr parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f065a00e18b59bbb015cf84fa1a5484d with result from task 645 parsl.dataflow.dflow DEBUG: Task 645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/33897794-2ef6-4888-afe1-913e4776c10b_isrForCrosstalkSources_1125899_28.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/33897794-2ef6-4888-afe1-913e4776c10b_isrForCrosstalkSources_1125899_28.stderr parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86d484104eb67e077a2b26f9546307a7 with result from task 653 parsl.dataflow.dflow DEBUG: Task 653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/4cbf1b39-b1c5-454c-b097-9b87f6c39af0_isrForCrosstalkSources_1125944_48.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/4cbf1b39-b1c5-454c-b097-9b87f6c39af0_isrForCrosstalkSources_1125944_48.stderr parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 939eb5c83864383f13263eed1d2f926a with result from task 665 parsl.dataflow.dflow DEBUG: Task 665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/aefb86fe-7e81-4a87-9036-8f8dc5a07114_isrForCrosstalkSources_1125922_35.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/aefb86fe-7e81-4a87-9036-8f8dc5a07114_isrForCrosstalkSources_1125922_35.stderr parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72cb71a8fef6e82a3166c2e935966995 with result from task 660 parsl.dataflow.dflow DEBUG: Task 660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/01252630-c733-4eb9-a4e6-a1a910098fa3_isrForCrosstalkSources_1125932_52.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/01252630-c733-4eb9-a4e6-a1a910098fa3_isrForCrosstalkSources_1125932_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:53 proc_lsst.multi:146 [INFO] found job 16896626 in provider checkpoint proc_lsst.multi INFO: found job 16896626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f876ddda257134a9a5b358535fab1f5 with result from task 657 parsl.dataflow.dflow DEBUG: Task 657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/8cb1ef68-1d87-4500-b2d1-d36e2dda1d3e_isrForCrosstalkSources_1125901_58.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/8cb1ef68-1d87-4500-b2d1-d36e2dda1d3e_isrForCrosstalkSources_1125901_58.stderr parsl.dataflow.dflow INFO: Task 641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fe3416039cedb4a1d176168791ca88c with result from task 641 parsl.dataflow.dflow DEBUG: Task 641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7ec64d66-ce0b-4ea9-9936-519d1ab2afe7_isrForCrosstalkSources_1125929_29.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7ec64d66-ce0b-4ea9-9936-519d1ab2afe7_isrForCrosstalkSources_1125929_29.stderr parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a762c9fa46d3f53b2c88c23ee246afa1 with result from task 643 parsl.dataflow.dflow DEBUG: Task 643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/c0e25158-f739-4be2-8ac7-bece00cd2617_isrForCrosstalkSources_1125942_52.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/c0e25158-f739-4be2-8ac7-bece00cd2617_isrForCrosstalkSources_1125942_52.stderr parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a90129749dca003018cba0aa1a5a6d6e with result from task 637 parsl.dataflow.dflow DEBUG: Task 637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/694a26bb-3478-4afd-a6ce-752bb4b24d4c_isrForCrosstalkSources_1125900_50.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/694a26bb-3478-4afd-a6ce-752bb4b24d4c_isrForCrosstalkSources_1125900_50.stderr parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8094c6500fbdbff5c126e78cd4f5a6e7 with result from task 651 parsl.dataflow.dflow DEBUG: Task 651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a6acc7d5-e51e-4973-8de4-1cdb48ee99a0_isrForCrosstalkSources_1125900_31.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a6acc7d5-e51e-4973-8de4-1cdb48ee99a0_isrForCrosstalkSources_1125900_31.stderr parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5178efa5e99013b10303d435de5b89b with result from task 649 parsl.dataflow.dflow DEBUG: Task 649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/da5a024f-92a0-4d53-984a-00743d03fa90_isrForCrosstalkSources_1125924_12.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/da5a024f-92a0-4d53-984a-00743d03fa90_isrForCrosstalkSources_1125924_12.stderr parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a7b7eea5713f685767f3f1b996cb72f with result from task 647 parsl.dataflow.dflow DEBUG: Task 647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/a0b04d21-fd02-4960-999d-0635cdf23a2f_isrForCrosstalkSources_1125946_48.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/a0b04d21-fd02-4960-999d-0635cdf23a2f_isrForCrosstalkSources_1125946_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:54 proc_lsst.multi:146 [INFO] found job 16896627 in provider checkpoint proc_lsst.multi INFO: found job 16896627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d1e15e9e8249582d1afcf438cbfcc4d with result from task 648 parsl.dataflow.dflow DEBUG: Task 648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/b01d9c90-fc8f-4f52-8f28-7df20ac76aa2_isrForCrosstalkSources_1125903_30.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/b01d9c90-fc8f-4f52-8f28-7df20ac76aa2_isrForCrosstalkSources_1125903_30.stderr parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54fb7f0a3cfcb5595260bfe211025198 with result from task 638 parsl.dataflow.dflow DEBUG: Task 638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/6a04c0e9-fbec-4357-998a-463db64eaa7d_isrForCrosstalkSources_1125917_43.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/6a04c0e9-fbec-4357-998a-463db64eaa7d_isrForCrosstalkSources_1125917_43.stderr parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae5f27c705e1cb7a2aa05ab6a2ba6b0a with result from task 667 parsl.dataflow.dflow DEBUG: Task 667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/e272eea1-188a-492e-a49e-f0a6b7bafbdf_isrForCrosstalkSources_1125938_37.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/e272eea1-188a-492e-a49e-f0a6b7bafbdf_isrForCrosstalkSources_1125938_37.stderr parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96ba927ecaca24b724b59849c6901e3e with result from task 666 parsl.dataflow.dflow DEBUG: Task 666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/907bdbba-245b-460a-8cfc-f5384f3dd506_isrForCrosstalkSources_1125922_17.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/907bdbba-245b-460a-8cfc-f5384f3dd506_isrForCrosstalkSources_1125922_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:54 proc_lsst.multi:146 [INFO] found job 16896628 in provider checkpoint proc_lsst.multi INFO: found job 16896628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5aa07423499a6ef22a378371a31cbee with result from task 663 parsl.dataflow.dflow DEBUG: Task 663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/723b9f26-2b4e-418b-808f-4d3f8b4398ff_isrForCrosstalkSources_1125917_1.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/723b9f26-2b4e-418b-808f-4d3f8b4398ff_isrForCrosstalkSources_1125917_1.stderr parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0573d41e0af18d21453587fc98410e67 with result from task 668 parsl.dataflow.dflow DEBUG: Task 668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/48945244-3b3a-4f7b-b389-3b523c8bc99f_isrForCrosstalkSources_1125937_36.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/48945244-3b3a-4f7b-b389-3b523c8bc99f_isrForCrosstalkSources_1125937_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:55 proc_lsst.multi:146 [INFO] found job 16896629 in provider checkpoint proc_lsst.multi INFO: found job 16896629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51536b4aa3a43c5a181880aa9f73e243 with result from task 670 parsl.dataflow.dflow DEBUG: Task 670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/db13a9a6-420f-4a62-9de0-1a9011b89246_isrForCrosstalkSources_1125919_41.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/db13a9a6-420f-4a62-9de0-1a9011b89246_isrForCrosstalkSources_1125919_41.stderr parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c936802fdb96717fb0c441d4d5c78332 with result from task 669 parsl.dataflow.dflow DEBUG: Task 669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/895927f3-3524-4693-a0f0-3bc9fb1b90d1_isrForCrosstalkSources_1125906_30.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/895927f3-3524-4693-a0f0-3bc9fb1b90d1_isrForCrosstalkSources_1125906_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:55 proc_lsst.multi:146 [INFO] found job 16896630 in provider checkpoint proc_lsst.multi INFO: found job 16896630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 477bebd7b47e7f5e065c993d9e744d6a with result from task 672 parsl.dataflow.dflow DEBUG: Task 672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/943df236-e5c3-45d2-9d6c-b381729c76df_isrForCrosstalkSources_1125941_5.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/943df236-e5c3-45d2-9d6c-b381729c76df_isrForCrosstalkSources_1125941_5.stderr parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45f49f171fe777523a1a5d37b0156777 with result from task 671 parsl.dataflow.dflow DEBUG: Task 671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/3cd5944c-c2ad-4f00-b3b9-53158e922d83_isrForCrosstalkSources_1125939_45.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/3cd5944c-c2ad-4f00-b3b9-53158e922d83_isrForCrosstalkSources_1125939_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:55 proc_lsst.multi:146 [INFO] found job 16896631 in provider checkpoint proc_lsst.multi INFO: found job 16896631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc200b4d219c4502438b0ef6bcdf6738 with result from task 675 parsl.dataflow.dflow DEBUG: Task 675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/c6d9044a-6b1b-4451-8c52-2ca0e54e22e6_isrForCrosstalkSources_1125925_3.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/c6d9044a-6b1b-4451-8c52-2ca0e54e22e6_isrForCrosstalkSources_1125925_3.stderr parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bcbfb47c9fa824fd4a2554d1404b754 with result from task 674 parsl.dataflow.dflow DEBUG: Task 674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/e1ca9711-c08e-4ab9-b27b-edddd36e8f91_isrForCrosstalkSources_1125914_11.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/e1ca9711-c08e-4ab9-b27b-edddd36e8f91_isrForCrosstalkSources_1125914_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:56 proc_lsst.multi:146 [INFO] found job 16896632 in provider checkpoint proc_lsst.multi INFO: found job 16896632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a85c9a043d6d9b171569552255ab4432 with result from task 673 parsl.dataflow.dflow DEBUG: Task 673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f4b54922-192b-4ffd-8832-7363a7d7e570_isrForCrosstalkSources_1125922_29.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f4b54922-192b-4ffd-8832-7363a7d7e570_isrForCrosstalkSources_1125922_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:56 proc_lsst.multi:146 [INFO] found job 16896633 in provider checkpoint proc_lsst.multi INFO: found job 16896633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 682fd575ef5ba9287590931cc2106340 with result from task 676 parsl.dataflow.dflow DEBUG: Task 676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/24174e74-97aa-4fa7-9ff4-f545d7a5e863_isrForCrosstalkSources_1125922_32.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/24174e74-97aa-4fa7-9ff4-f545d7a5e863_isrForCrosstalkSources_1125922_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:57 proc_lsst.multi:146 [INFO] found job 16896634 in provider checkpoint proc_lsst.multi INFO: found job 16896634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:57 proc_lsst.multi:146 [INFO] found job 16896635 in provider checkpoint proc_lsst.multi INFO: found job 16896635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:58 proc_lsst.multi:146 [INFO] found job 16896636 in provider checkpoint proc_lsst.multi INFO: found job 16896636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:59 proc_lsst.multi:146 [INFO] found job 16896637 in provider checkpoint proc_lsst.multi INFO: found job 16896637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f00993645aa3a322d6e118809bf0e4fd with result from task 683 parsl.dataflow.dflow DEBUG: Task 683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/50dcdd19-d00a-4cd1-a373-e4724cde3f12_isrForCrosstalkSources_1125905_48.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/50dcdd19-d00a-4cd1-a373-e4724cde3f12_isrForCrosstalkSources_1125905_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:10:59 proc_lsst.multi:146 [INFO] found job 16896638 in provider checkpoint proc_lsst.multi INFO: found job 16896638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:00 proc_lsst.multi:146 [INFO] found job 16896639 in provider checkpoint proc_lsst.multi INFO: found job 16896639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:00 proc_lsst.multi:146 [INFO] found job 16896640 in provider checkpoint proc_lsst.multi INFO: found job 16896640 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:01 proc_lsst.multi:146 [INFO] found job 16896641 in provider checkpoint proc_lsst.multi INFO: found job 16896641 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c567873298f67a2a82c76387b45a31f with result from task 705 parsl.dataflow.dflow DEBUG: Task 705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/d00a3b80-6683-466b-865d-d8732fb6f245_isrForCrosstalkSources_1125897_6.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/d00a3b80-6683-466b-865d-d8732fb6f245_isrForCrosstalkSources_1125897_6.stderr parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 933ac74a0796bc0fd2e4de0e75a04743 with result from task 700 parsl.dataflow.dflow DEBUG: Task 700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/a829ba6f-ef82-4805-808d-eed6c28a257e_isrForCrosstalkSources_1125927_13.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/a829ba6f-ef82-4805-808d-eed6c28a257e_isrForCrosstalkSources_1125927_13.stderr parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a2d5533a49c911e4f2687bf67eace9c with result from task 677 parsl.dataflow.dflow DEBUG: Task 677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/4db11ff5-21c4-44c7-8797-fd5fe49e3f5a_isrForCrosstalkSources_1125934_34.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/4db11ff5-21c4-44c7-8797-fd5fe49e3f5a_isrForCrosstalkSources_1125934_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:01 proc_lsst.multi:146 [INFO] found job 16896642 in provider checkpoint proc_lsst.multi INFO: found job 16896642 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fe394745626d0d9b820086b822e92c0 with result from task 678 parsl.dataflow.dflow DEBUG: Task 678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/7956e711-9ec7-4e9e-abee-e869780ed09a_isrForCrosstalkSources_1125892_13.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/7956e711-9ec7-4e9e-abee-e869780ed09a_isrForCrosstalkSources_1125892_13.stderr parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3246698bb058862250445766d76c132 with result from task 709 parsl.dataflow.dflow DEBUG: Task 709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/f6b21adb-64ae-4c80-b45f-9800069fe19a_isrForCrosstalkSources_1125918_24.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/f6b21adb-64ae-4c80-b45f-9800069fe19a_isrForCrosstalkSources_1125918_24.stderr parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a6def2eafc86e8c1b37c2ab53e53e1c with result from task 695 parsl.dataflow.dflow DEBUG: Task 695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/1539bc16-1755-477a-a312-82a17488c9ef_isrForCrosstalkSources_1125942_62.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/1539bc16-1755-477a-a312-82a17488c9ef_isrForCrosstalkSources_1125942_62.stderr parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 655a4185d41bd3a79b799d8774a7cff5 with result from task 694 parsl.dataflow.dflow DEBUG: Task 694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/4fc6af07-92de-4f2c-8f6f-ffb751f3cc4f_isrForCrosstalkSources_1125942_22.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/4fc6af07-92de-4f2c-8f6f-ffb751f3cc4f_isrForCrosstalkSources_1125942_22.stderr parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d241dabd5251eee05a4cbddb7f20cca9 with result from task 679 parsl.dataflow.dflow DEBUG: Task 679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/dff8a3df-4fdc-4500-97b3-49d3ea823d72_isrForCrosstalkSources_1125946_33.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/dff8a3df-4fdc-4500-97b3-49d3ea823d72_isrForCrosstalkSources_1125946_33.stderr parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bce28671393b1c12c77c14fbf30b9d33 with result from task 704 parsl.dataflow.dflow DEBUG: Task 704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/c67d914d-8a2d-488b-ac07-4e4cf9152146_isrForCrosstalkSources_1125945_21.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/c67d914d-8a2d-488b-ac07-4e4cf9152146_isrForCrosstalkSources_1125945_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:01 proc_lsst.multi:146 [INFO] found job 16896643 in provider checkpoint proc_lsst.multi INFO: found job 16896643 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e6e5257b5a873f6eab341148c2257f7 with result from task 693 parsl.dataflow.dflow DEBUG: Task 693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/f4691488-7fa0-4bb8-9c67-f1cfa66ba7c6_isrForCrosstalkSources_1125899_4.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/f4691488-7fa0-4bb8-9c67-f1cfa66ba7c6_isrForCrosstalkSources_1125899_4.stderr parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 133d02467ed589a1d43c45637e7e0b8f with result from task 701 parsl.dataflow.dflow DEBUG: Task 701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/3caef4ad-6e3d-4965-9e88-edf215a19a98_isrForCrosstalkSources_1125891_14.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/3caef4ad-6e3d-4965-9e88-edf215a19a98_isrForCrosstalkSources_1125891_14.stderr parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b47544a9aceb3c5e3870cefdf9c1aa7 with result from task 716 parsl.dataflow.dflow DEBUG: Task 716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/2f4a1cec-6c8b-49e1-8b92-b1f845ebd6d5_isrForCrosstalkSources_1125947_26.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/2f4a1cec-6c8b-49e1-8b92-b1f845ebd6d5_isrForCrosstalkSources_1125947_26.stderr parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c3e595b5a6b5955df3e285b85c53c40 with result from task 692 parsl.dataflow.dflow DEBUG: Task 692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/95071f16-2744-42fd-8f5e-db3a18d04db4_isrForCrosstalkSources_1125903_38.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/95071f16-2744-42fd-8f5e-db3a18d04db4_isrForCrosstalkSources_1125903_38.stderr parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ced73ce03545586b9b4b61d7581a4a77 with result from task 697 parsl.dataflow.dflow DEBUG: Task 697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/dbb6769e-cc60-4696-94f2-3644c7a810e1_isrForCrosstalkSources_1125905_33.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/dbb6769e-cc60-4696-94f2-3644c7a810e1_isrForCrosstalkSources_1125905_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:02 proc_lsst.multi:146 [INFO] found job 16896644 in provider checkpoint proc_lsst.multi INFO: found job 16896644 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ec62e122d554c4a48e19ed9da148217 with result from task 699 parsl.dataflow.dflow DEBUG: Task 699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/42d543c7-2c52-4e1a-9762-7923c45274ed_isrForCrosstalkSources_1125945_53.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/42d543c7-2c52-4e1a-9762-7923c45274ed_isrForCrosstalkSources_1125945_53.stderr parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 794b68e27cccb50251f38ed2e63996e2 with result from task 698 parsl.dataflow.dflow DEBUG: Task 698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/d3a172fc-9c41-4ede-bcbc-c44bc8458ea7_isrForCrosstalkSources_1125945_30.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/d3a172fc-9c41-4ede-bcbc-c44bc8458ea7_isrForCrosstalkSources_1125945_30.stderr parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 885c2b7b2c409fe3ed17dd92a58b5d0d with result from task 703 parsl.dataflow.dflow DEBUG: Task 703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/4b3d29cf-db80-4e90-a972-ebc14870513d_isrForCrosstalkSources_1125891_12.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/4b3d29cf-db80-4e90-a972-ebc14870513d_isrForCrosstalkSources_1125891_12.stderr parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8673ddc87896fed23fc3253b53420540 with result from task 691 parsl.dataflow.dflow DEBUG: Task 691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/4d8bf06b-f796-4c95-9608-af680c4ca103_isrForCrosstalkSources_1125901_61.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/4d8bf06b-f796-4c95-9608-af680c4ca103_isrForCrosstalkSources_1125901_61.stderr parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9326af715963c042b51b2ef84751a87 with result from task 707 parsl.dataflow.dflow DEBUG: Task 707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/dc8432fc-2513-4322-acb7-c42a22b08c8e_isrForCrosstalkSources_1125933_8.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/dc8432fc-2513-4322-acb7-c42a22b08c8e_isrForCrosstalkSources_1125933_8.stderr parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a2c4af316322e944f496fa769a40092 with result from task 690 parsl.dataflow.dflow DEBUG: Task 690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/fe921c47-c2cb-47fd-a254-bc5575094594_isrForCrosstalkSources_1125930_16.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/fe921c47-c2cb-47fd-a254-bc5575094594_isrForCrosstalkSources_1125930_16.stderr parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e5732c0771a55540225c7b8e5359cc7 with result from task 685 parsl.dataflow.dflow DEBUG: Task 685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/6c251bf5-890d-4ba1-a4f8-f53f1073688f_isrForCrosstalkSources_1125921_17.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/6c251bf5-890d-4ba1-a4f8-f53f1073688f_isrForCrosstalkSources_1125921_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:02 proc_lsst.multi:146 [INFO] found job 16896645 in provider checkpoint proc_lsst.multi INFO: found job 16896645 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3252e0e17a065dccd7c21fcdfa7e311e with result from task 682 parsl.dataflow.dflow DEBUG: Task 682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/13f53b0a-52a2-42fa-875f-7e02873e4ddb_isrForCrosstalkSources_1125913_12.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/13f53b0a-52a2-42fa-875f-7e02873e4ddb_isrForCrosstalkSources_1125913_12.stderr parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aaa82791ca58242ce00ace1f29ab4f66 with result from task 684 parsl.dataflow.dflow DEBUG: Task 684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d220fdbf-a227-44f4-a256-9cf326e7dddc_isrForCrosstalkSources_1125940_11.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d220fdbf-a227-44f4-a256-9cf326e7dddc_isrForCrosstalkSources_1125940_11.stderr parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70251b62c50e9aeddc9e341aeb9a6d7e with result from task 702 parsl.dataflow.dflow DEBUG: Task 702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/a611dd9e-23ba-4b1a-96cf-3031df86acd6_isrForCrosstalkSources_1125888_29.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/a611dd9e-23ba-4b1a-96cf-3031df86acd6_isrForCrosstalkSources_1125888_29.stderr parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11670301a704586499837a71832068bc with result from task 689 parsl.dataflow.dflow DEBUG: Task 689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/991914a0-f974-4f14-8cce-767aa4da6cf2_isrForCrosstalkSources_1125932_58.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/991914a0-f974-4f14-8cce-767aa4da6cf2_isrForCrosstalkSources_1125932_58.stderr parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8929b47efd7bd37cc1582993b671ba4 with result from task 712 parsl.dataflow.dflow DEBUG: Task 712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/84fa837c-52f9-4216-8ce5-224a83613507_isrForCrosstalkSources_1125930_27.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/84fa837c-52f9-4216-8ce5-224a83613507_isrForCrosstalkSources_1125930_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:03 proc_lsst.multi:146 [INFO] found job 16896646 in provider checkpoint proc_lsst.multi INFO: found job 16896646 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56d059cd348e7e7c5f5bbdf0c5d784ca with result from task 710 parsl.dataflow.dflow DEBUG: Task 710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/00732f5e-95ba-466b-afd8-f81b4a7cd27a_isrForCrosstalkSources_1125891_34.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/00732f5e-95ba-466b-afd8-f81b4a7cd27a_isrForCrosstalkSources_1125891_34.stderr parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 590acdc768ce4ab48d62b8f7ebeea696 with result from task 706 parsl.dataflow.dflow DEBUG: Task 706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2f756ded-94d8-404f-9307-6c565df5e351_isrForCrosstalkSources_1125895_6.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2f756ded-94d8-404f-9307-6c565df5e351_isrForCrosstalkSources_1125895_6.stderr parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d449055026f8a92f1a1b8a10b5698b59 with result from task 688 parsl.dataflow.dflow DEBUG: Task 688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/d20c8fb1-4fd9-4e18-bc40-78819570b4c1_isrForCrosstalkSources_1125889_26.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/d20c8fb1-4fd9-4e18-bc40-78819570b4c1_isrForCrosstalkSources_1125889_26.stderr parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de4c670a335b04d736b741a3ae549f08 with result from task 681 parsl.dataflow.dflow DEBUG: Task 681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/227673b5-68c0-43e1-b245-3ec5a54fc69d_isrForCrosstalkSources_1125919_28.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/227673b5-68c0-43e1-b245-3ec5a54fc69d_isrForCrosstalkSources_1125919_28.stderr parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3caf66a86c6cab2f56440d2bf8aeeb57 with result from task 696 parsl.dataflow.dflow DEBUG: Task 696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a6845c39-e197-4ddf-b224-0e91f647f6a3_isrForCrosstalkSources_1125933_43.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a6845c39-e197-4ddf-b224-0e91f647f6a3_isrForCrosstalkSources_1125933_43.stderr parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 040a0bbaef90ebdaa412299003d9f094 with result from task 687 parsl.dataflow.dflow DEBUG: Task 687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/f20cff46-4925-4f30-a696-24e9238eabf8_isrForCrosstalkSources_1125935_28.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/f20cff46-4925-4f30-a696-24e9238eabf8_isrForCrosstalkSources_1125935_28.stderr parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 895e4c1401cb68d7ba2533c44680577f with result from task 708 parsl.dataflow.dflow DEBUG: Task 708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/746b81b9-9e59-46da-949b-4b5e5e920ee7_isrForCrosstalkSources_1125916_12.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/746b81b9-9e59-46da-949b-4b5e5e920ee7_isrForCrosstalkSources_1125916_12.stderr parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16cb7020933bbfcb92f48975f8635a98 with result from task 711 parsl.dataflow.dflow DEBUG: Task 711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/106ba483-46f6-402b-8144-649a2fe6047f_isrForCrosstalkSources_1125932_42.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/106ba483-46f6-402b-8144-649a2fe6047f_isrForCrosstalkSources_1125932_42.stderr parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 998b0c337001affa4cad67702c938246 with result from task 715 parsl.dataflow.dflow DEBUG: Task 715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/1983f850-10d5-402a-ae63-194dc2fc048e_isrForCrosstalkSources_1125931_33.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/1983f850-10d5-402a-ae63-194dc2fc048e_isrForCrosstalkSources_1125931_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:04 proc_lsst.multi:146 [INFO] found job 16896647 in provider checkpoint proc_lsst.multi INFO: found job 16896647 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d9741e0a18a7b6d23c1c2ffcb44750d with result from task 714 parsl.dataflow.dflow DEBUG: Task 714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/345576a6-1450-4a7c-b315-3e930c0bb1f4_isrForCrosstalkSources_1125934_33.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/345576a6-1450-4a7c-b315-3e930c0bb1f4_isrForCrosstalkSources_1125934_33.stderr parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 470c1e2b9e468e7cd91405b636b7e447 with result from task 713 parsl.dataflow.dflow DEBUG: Task 713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/11c55d90-ec2c-4058-a7e7-3617a610e318_isrForCrosstalkSources_1125933_40.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/11c55d90-ec2c-4058-a7e7-3617a610e318_isrForCrosstalkSources_1125933_40.stderr parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 912a7ceeab380beb2e6f5f25fb4f6f91 with result from task 717 parsl.dataflow.dflow DEBUG: Task 717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/420bde2e-bc97-4557-bd52-38ab3ac01b21_isrForCrosstalkSources_1125915_44.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/420bde2e-bc97-4557-bd52-38ab3ac01b21_isrForCrosstalkSources_1125915_44.stderr parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7af4c80b4df61512caf6aba3329e5842 with result from task 720 parsl.dataflow.dflow DEBUG: Task 720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/fe8b18be-2a7a-42c1-bf82-e5748a4b2fa3_isrForCrosstalkSources_1125901_7.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/fe8b18be-2a7a-42c1-bf82-e5748a4b2fa3_isrForCrosstalkSources_1125901_7.stderr parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5cbdc3dac4a050fc5cb0a9c6f9c77d1 with result from task 719 parsl.dataflow.dflow DEBUG: Task 719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/ff8ed4d2-9b77-48d5-973d-30638a5e9f04_isrForCrosstalkSources_1125917_44.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/ff8ed4d2-9b77-48d5-973d-30638a5e9f04_isrForCrosstalkSources_1125917_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:04 proc_lsst.multi:146 [INFO] found job 16896648 in provider checkpoint proc_lsst.multi INFO: found job 16896648 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29afc7ca1a7ea7028748a35ba2c49110 with result from task 721 parsl.dataflow.dflow DEBUG: Task 721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/b3c045c2-00f6-4f12-b188-c124bf3eed30_isrForCrosstalkSources_1125908_16.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/b3c045c2-00f6-4f12-b188-c124bf3eed30_isrForCrosstalkSources_1125908_16.stderr parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 452d67cbc1db36352de9363abb73081b with result from task 680 parsl.dataflow.dflow DEBUG: Task 680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/aebb9ff5-5efa-4fed-a117-240daa5a061a_isrForCrosstalkSources_1125906_50.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/aebb9ff5-5efa-4fed-a117-240daa5a061a_isrForCrosstalkSources_1125906_50.stderr parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 672dee9745f06e49498841ec3b2b71a7 with result from task 723 parsl.dataflow.dflow DEBUG: Task 723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/fabc9f51-f697-4cd1-b62f-e1b096491f24_isrForCrosstalkSources_1125900_20.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/fabc9f51-f697-4cd1-b62f-e1b096491f24_isrForCrosstalkSources_1125900_20.stderr parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 933f120b6ddce53474f79ca34c6aa7a0 with result from task 722 parsl.dataflow.dflow DEBUG: Task 722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/83550407-ee95-472b-9dce-0e83a1c72c77_isrForCrosstalkSources_1125904_9.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/83550407-ee95-472b-9dce-0e83a1c72c77_isrForCrosstalkSources_1125904_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:05 proc_lsst.multi:146 [INFO] found job 16896649 in provider checkpoint proc_lsst.multi INFO: found job 16896649 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8a3353b1e6c7ab180e8e7cc5a624f01 with result from task 725 parsl.dataflow.dflow DEBUG: Task 725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/99c2d408-48ec-4366-9524-8312e0c56a9e_isrForCrosstalkSources_1125906_52.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/99c2d408-48ec-4366-9524-8312e0c56a9e_isrForCrosstalkSources_1125906_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:05 proc_lsst.multi:146 [INFO] found job 16896650 in provider checkpoint proc_lsst.multi INFO: found job 16896650 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a293af78f12d05f1444ae6cf650f991 with result from task 724 parsl.dataflow.dflow DEBUG: Task 724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/97ade5a3-2ed0-4d4b-af06-ebc2bde4db6f_isrForCrosstalkSources_1125900_62.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/97ade5a3-2ed0-4d4b-af06-ebc2bde4db6f_isrForCrosstalkSources_1125900_62.stderr parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00fadabb1ef07c572577d172a626fcbe with result from task 686 parsl.dataflow.dflow DEBUG: Task 686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/5ff75467-3c29-4b7b-acbb-2f6d4c6769c3_isrForCrosstalkSources_1125946_5.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/5ff75467-3c29-4b7b-acbb-2f6d4c6769c3_isrForCrosstalkSources_1125946_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:06 proc_lsst.multi:146 [INFO] found job 16896651 in provider checkpoint proc_lsst.multi INFO: found job 16896651 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a79c9ba522c71f47298f531afa231a53 with result from task 726 parsl.dataflow.dflow DEBUG: Task 726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/dac2beb3-cf94-4bbd-86bf-4a93f1a827d8_isrForCrosstalkSources_1125929_20.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/dac2beb3-cf94-4bbd-86bf-4a93f1a827d8_isrForCrosstalkSources_1125929_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:06 proc_lsst.multi:146 [INFO] found job 16896652 in provider checkpoint proc_lsst.multi INFO: found job 16896652 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6546daf2cf71c129004425c7add1640f with result from task 727 parsl.dataflow.dflow DEBUG: Task 727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/5b0bcb2e-def8-42e5-9368-d9e5a07a8eb9_isrForCrosstalkSources_1125936_9.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/5b0bcb2e-def8-42e5-9368-d9e5a07a8eb9_isrForCrosstalkSources_1125936_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:07 proc_lsst.multi:146 [INFO] found job 16896653 in provider checkpoint proc_lsst.multi INFO: found job 16896653 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d833303a387f5cc2f93a81f4d1cafbcb with result from task 718 parsl.dataflow.dflow DEBUG: Task 718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/dda2c83d-ed7d-4b26-9281-512677e84c84_isrForCrosstalkSources_1125939_49.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/dda2c83d-ed7d-4b26-9281-512677e84c84_isrForCrosstalkSources_1125939_49.stderr parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcee2582da20a55fd1b629106dea3dd7 with result from task 728 parsl.dataflow.dflow DEBUG: Task 728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/a06249a3-efc8-4b2d-81e2-856115be8c6c_isrForCrosstalkSources_1125902_8.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/a06249a3-efc8-4b2d-81e2-856115be8c6c_isrForCrosstalkSources_1125902_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:07 proc_lsst.multi:146 [INFO] found job 16896654 in provider checkpoint proc_lsst.multi INFO: found job 16896654 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:08 proc_lsst.multi:146 [INFO] found job 16896655 in provider checkpoint proc_lsst.multi INFO: found job 16896655 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:08 proc_lsst.multi:146 [INFO] found job 16896656 in provider checkpoint proc_lsst.multi INFO: found job 16896656 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4b73111f7222eeb8de98fd9a911aaa1 with result from task 729 parsl.dataflow.dflow DEBUG: Task 729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/238c8414-cd5d-48eb-aa93-d0e030917ea1_isrForCrosstalkSources_1125941_52.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/238c8414-cd5d-48eb-aa93-d0e030917ea1_isrForCrosstalkSources_1125941_52.stderr parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef61a30ca3b00d4fd84b8b603ced0538 with result from task 740 parsl.dataflow.dflow DEBUG: Task 740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/00db2054-d53b-4313-9f38-4bffe912ed86_isrForCrosstalkSources_1125888_20.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/00db2054-d53b-4313-9f38-4bffe912ed86_isrForCrosstalkSources_1125888_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD 16896473 PD 16896472 PD 16896471 PD 16896470 PD 16896469 PD 16896468 PD 16896467 PD 16896466 PD 16896464 PD 16896463 PD 16896462 PD 16896461 PD 16896460 PD 16896459 PD 16896458 PD 16896457 PD 16896456 PD 16896455 PD 16896454 PD 16896453 PD 16896452 PD 16896451 PD 16896450 PD 16896449 PD 16896448 PD 16896447 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:11 proc_lsst.multi:146 [INFO] found job 16896657 in provider checkpoint proc_lsst.multi INFO: found job 16896657 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c05bf56b6f4b506b1eaf81199a9b196d with result from task 742 parsl.dataflow.dflow DEBUG: Task 742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/18b24264-0d32-4cde-8f49-6eafba851f25_isrForCrosstalkSources_1125905_34.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/18b24264-0d32-4cde-8f49-6eafba851f25_isrForCrosstalkSources_1125905_34.stderr parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b88bfad8e7918304a6c2782bc8e60ae with result from task 743 parsl.dataflow.dflow DEBUG: Task 743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/ab44e016-3097-4c46-a666-6b5f2279c3c8_isrForCrosstalkSources_1125888_35.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/ab44e016-3097-4c46-a666-6b5f2279c3c8_isrForCrosstalkSources_1125888_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896475 PD 16896474 PD 16896473 PD 16896472 PD 16896471 PD 16896470 PD 16896469 PD 16896468 PD 16896467 PD 16896466 PD 16896464 PD 16896463 PD 16896462 PD 16896461 PD 16896460 PD 16896459 PD 16896458 PD 16896457 PD 16896456 PD 16896455 PD 16896454 PD 16896453 PD 16896452 PD 16896451 PD 16896450 PD 16896449 PD 16896448 PD 16896447 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:11 proc_lsst.multi:146 [INFO] found job 16896658 in provider checkpoint proc_lsst.multi INFO: found job 16896658 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9cd9a3b12a5b2566695f3f8e510868f with result from task 734 parsl.dataflow.dflow DEBUG: Task 734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/3cb2e033-a0d4-4b24-b893-c0bb7e664ce6_isrForCrosstalkSources_1125927_26.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/3cb2e033-a0d4-4b24-b893-c0bb7e664ce6_isrForCrosstalkSources_1125927_26.stderr parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d36a9da0d7c1aa1877269d77a8c7f3bb with result from task 736 parsl.dataflow.dflow DEBUG: Task 736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/1f26c640-e843-47c9-8160-12a592b9a7cc_isrForCrosstalkSources_1125913_56.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/1f26c640-e843-47c9-8160-12a592b9a7cc_isrForCrosstalkSources_1125913_56.stderr parsl.dataflow.dflow INFO: Task 747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ebb39eae9293369ed1ce2fbb2528162 with result from task 747 parsl.dataflow.dflow DEBUG: Task 747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/0a7d360e-eedb-4dde-beed-00931e1e3737_isrForCrosstalkSources_1125943_8.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/0a7d360e-eedb-4dde-beed-00931e1e3737_isrForCrosstalkSources_1125943_8.stderr parsl.dataflow.dflow INFO: Task 751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d9ff323c1df5083b03b9b16f656a35e with result from task 751 parsl.dataflow.dflow DEBUG: Task 751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/e0d3571a-e7cf-4f32-bad2-87516f494c69_isrForCrosstalkSources_1125900_40.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/e0d3571a-e7cf-4f32-bad2-87516f494c69_isrForCrosstalkSources_1125900_40.stderr parsl.dataflow.dflow INFO: Task 744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39c91d9f4810fdf424d05210e9c57b83 with result from task 744 parsl.dataflow.dflow DEBUG: Task 744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/a2eff311-13e3-4f23-920b-55f78e238ce7_isrForCrosstalkSources_1125911_45.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/a2eff311-13e3-4f23-920b-55f78e238ce7_isrForCrosstalkSources_1125911_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:12 proc_lsst.multi:146 [INFO] found job 16896659 in provider checkpoint proc_lsst.multi INFO: found job 16896659 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cebbfbb9d428b33f56122f85a1013d11 with result from task 738 parsl.dataflow.dflow DEBUG: Task 738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/6295614f-9241-4bbb-a8d9-8540426e082a_isrForCrosstalkSources_1125899_57.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/6295614f-9241-4bbb-a8d9-8540426e082a_isrForCrosstalkSources_1125899_57.stderr parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e080894ecab500bcb5dc7a60bb5e7a2 with result from task 739 parsl.dataflow.dflow DEBUG: Task 739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/5d2ef4f0-a0f1-4fc9-9d05-c84efdeca7bb_isrForCrosstalkSources_1125938_4.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/5d2ef4f0-a0f1-4fc9-9d05-c84efdeca7bb_isrForCrosstalkSources_1125938_4.stderr parsl.dataflow.dflow INFO: Task 746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 924161e4fbf49cf30c259bb9c16443b5 with result from task 746 parsl.dataflow.dflow DEBUG: Task 746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/5046da32-6f73-44f4-9eac-95895e81de76_isrForCrosstalkSources_1125899_44.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/5046da32-6f73-44f4-9eac-95895e81de76_isrForCrosstalkSources_1125899_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:12 proc_lsst.multi:146 [INFO] found job 16896660 in provider checkpoint proc_lsst.multi INFO: found job 16896660 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ab3317277df8bde0135c48c0cf96293 with result from task 758 parsl.dataflow.dflow DEBUG: Task 758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/e771613c-402a-420d-8bc1-ef52922195a9_isrForCrosstalkSources_1125888_50.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/e771613c-402a-420d-8bc1-ef52922195a9_isrForCrosstalkSources_1125888_50.stderr parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e15ab56cc54ea2f74cb7a6b0f1188d85 with result from task 741 parsl.dataflow.dflow DEBUG: Task 741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/93bcd3b2-a189-4b7b-8ac3-138fc3164a1f_isrForCrosstalkSources_1125937_30.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/93bcd3b2-a189-4b7b-8ac3-138fc3164a1f_isrForCrosstalkSources_1125937_30.stderr parsl.dataflow.dflow INFO: Task 750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b1dfc5ca7e82e58a84df5172201d812 with result from task 750 parsl.dataflow.dflow DEBUG: Task 750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/e4cd3189-80cf-4588-ad23-b5137124b9b7_isrForCrosstalkSources_1125896_52.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/e4cd3189-80cf-4588-ad23-b5137124b9b7_isrForCrosstalkSources_1125896_52.stderr parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd46b280449f0f4aa8ff536ef205ded2 with result from task 735 parsl.dataflow.dflow DEBUG: Task 735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/4defcb48-6dee-4b68-b7b9-f3656aa46ac2_isrForCrosstalkSources_1125922_47.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/4defcb48-6dee-4b68-b7b9-f3656aa46ac2_isrForCrosstalkSources_1125922_47.stderr parsl.dataflow.dflow INFO: Task 759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fb28f9c67d71eb4a399930aff59de95 with result from task 759 parsl.dataflow.dflow DEBUG: Task 759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/7225b83b-d58c-4c0d-80d5-f67808fb65ca_isrForCrosstalkSources_1125912_41.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/7225b83b-d58c-4c0d-80d5-f67808fb65ca_isrForCrosstalkSources_1125912_41.stderr parsl.dataflow.dflow INFO: Task 752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9daf9d2b112d09aedfb6af93dea15500 with result from task 752 parsl.dataflow.dflow DEBUG: Task 752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/564dd0c4-21fe-45ef-b8f3-a2c5747796ae_isrForCrosstalkSources_1125931_31.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/564dd0c4-21fe-45ef-b8f3-a2c5747796ae_isrForCrosstalkSources_1125931_31.stderr parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a2037b4b37524a69d541e8bfbd798d1 with result from task 733 parsl.dataflow.dflow DEBUG: Task 733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/7c953b72-b776-420e-8d35-e6ddd92f1584_isrForCrosstalkSources_1125946_25.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/7c953b72-b776-420e-8d35-e6ddd92f1584_isrForCrosstalkSources_1125946_25.stderr parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 511fcb09af38cc2277507b78fdaadbb8 with result from task 731 parsl.dataflow.dflow DEBUG: Task 731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/22a00018-7afd-4e9d-ace6-1ba98560b115_isrForCrosstalkSources_1125945_27.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/22a00018-7afd-4e9d-ace6-1ba98560b115_isrForCrosstalkSources_1125945_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:13 proc_lsst.multi:146 [INFO] found job 16896661 in provider checkpoint proc_lsst.multi INFO: found job 16896661 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6349a01f13dfc7cbb39a05472515a680 with result from task 767 parsl.dataflow.dflow DEBUG: Task 767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/0801ea30-7ee2-4d21-bb73-632e152f5d54_isrForCrosstalkSources_1125895_20.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/0801ea30-7ee2-4d21-bb73-632e152f5d54_isrForCrosstalkSources_1125895_20.stderr parsl.dataflow.dflow INFO: Task 766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15944d1f86061c1e40158b0e562bbb9e with result from task 766 parsl.dataflow.dflow DEBUG: Task 766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/5e10e291-c105-4c88-aed5-659bf66a6205_isrForCrosstalkSources_1125890_20.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/5e10e291-c105-4c88-aed5-659bf66a6205_isrForCrosstalkSources_1125890_20.stderr parsl.dataflow.dflow INFO: Task 760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b8600de244ad9cf11d84b7fe5510b50 with result from task 760 parsl.dataflow.dflow DEBUG: Task 760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ba036c59-2567-48ee-805b-1a2b054cbeb5_isrForCrosstalkSources_1125918_61.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ba036c59-2567-48ee-805b-1a2b054cbeb5_isrForCrosstalkSources_1125918_61.stderr parsl.dataflow.dflow INFO: Task 755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7062f7176890e557b034f372585bd5f4 with result from task 755 parsl.dataflow.dflow DEBUG: Task 755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/53dd4eca-47ed-4bef-b362-0db4c00e86b4_isrForCrosstalkSources_1125895_40.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/53dd4eca-47ed-4bef-b362-0db4c00e86b4_isrForCrosstalkSources_1125895_40.stderr parsl.dataflow.dflow INFO: Task 749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5161517c4550ac4a66f645f0032810b with result from task 749 parsl.dataflow.dflow DEBUG: Task 749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/e0869e3f-0eaf-47af-8b4d-b1ad4332c87f_isrForCrosstalkSources_1125912_59.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/e0869e3f-0eaf-47af-8b4d-b1ad4332c87f_isrForCrosstalkSources_1125912_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:13 proc_lsst.multi:146 [INFO] found job 16896662 in provider checkpoint proc_lsst.multi INFO: found job 16896662 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68d26701620e357b6218df073df33ef6 with result from task 765 parsl.dataflow.dflow DEBUG: Task 765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/c048a176-5cf1-42c8-a614-47a7ba092fec_isrForCrosstalkSources_1125899_25.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/c048a176-5cf1-42c8-a614-47a7ba092fec_isrForCrosstalkSources_1125899_25.stderr parsl.dataflow.dflow INFO: Task 763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00f7c116433a9b2dcb662a6f3c84ee5c with result from task 763 parsl.dataflow.dflow DEBUG: Task 763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/0cdab88b-bbc3-4596-9043-fedccd6399a7_isrForCrosstalkSources_1125909_33.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/0cdab88b-bbc3-4596-9043-fedccd6399a7_isrForCrosstalkSources_1125909_33.stderr parsl.dataflow.dflow INFO: Task 754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 198f742486ca77ee7e33ba311260816b with result from task 754 parsl.dataflow.dflow DEBUG: Task 754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/c228f2b6-1e61-434f-acab-783f0e832b94_isrForCrosstalkSources_1125941_37.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/c228f2b6-1e61-434f-acab-783f0e832b94_isrForCrosstalkSources_1125941_37.stderr parsl.dataflow.dflow INFO: Task 770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eac7c6b6e88901919fc8656cde8fcbc0 with result from task 770 parsl.dataflow.dflow DEBUG: Task 770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/86a1c3b6-d12d-4c1b-9965-8164be7e256f_isrForCrosstalkSources_1125924_4.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/86a1c3b6-d12d-4c1b-9965-8164be7e256f_isrForCrosstalkSources_1125924_4.stderr parsl.dataflow.dflow INFO: Task 764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10bf248801c70bf8ae2c10fe49c97224 with result from task 764 parsl.dataflow.dflow DEBUG: Task 764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/9fd9e7e4-b70f-4e98-a0f1-4cea85849174_isrForCrosstalkSources_1125905_32.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/9fd9e7e4-b70f-4e98-a0f1-4cea85849174_isrForCrosstalkSources_1125905_32.stderr parsl.dataflow.dflow INFO: Task 761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6f8e639d2046867bca28d19e88f2e0b with result from task 761 parsl.dataflow.dflow DEBUG: Task 761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/b49db1c5-20eb-40ff-a001-23e05717be2e_isrForCrosstalkSources_1125895_55.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/b49db1c5-20eb-40ff-a001-23e05717be2e_isrForCrosstalkSources_1125895_55.stderr parsl.dataflow.dflow INFO: Task 769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 083697699cdcae61bf2d205eba8c6988 with result from task 769 parsl.dataflow.dflow DEBUG: Task 769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/5beec3d1-1034-44e1-b23c-e153bd809f42_isrForCrosstalkSources_1125930_8.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/5beec3d1-1034-44e1-b23c-e153bd809f42_isrForCrosstalkSources_1125930_8.stderr parsl.dataflow.dflow INFO: Task 745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ed8a5c888bde8b766d1d16552b71174 with result from task 745 parsl.dataflow.dflow DEBUG: Task 745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/e8a2d553-839a-49b0-be2c-308b670bd86e_isrForCrosstalkSources_1125893_60.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/e8a2d553-839a-49b0-be2c-308b670bd86e_isrForCrosstalkSources_1125893_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:14 proc_lsst.multi:146 [INFO] found job 16896663 in provider checkpoint proc_lsst.multi INFO: found job 16896663 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4118fae4ca942208a2016a615248205 with result from task 762 parsl.dataflow.dflow DEBUG: Task 762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/a7e3785c-554a-4877-94b0-9b5d08428278_isrForCrosstalkSources_1125920_21.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/a7e3785c-554a-4877-94b0-9b5d08428278_isrForCrosstalkSources_1125920_21.stderr parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5031da1e08d16284e270d32dc31512d9 with result from task 737 parsl.dataflow.dflow DEBUG: Task 737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/7febfae3-e83c-41c4-b992-f3ebcd5517da_isrForCrosstalkSources_1125905_58.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/7febfae3-e83c-41c4-b992-f3ebcd5517da_isrForCrosstalkSources_1125905_58.stderr parsl.dataflow.dflow INFO: Task 768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5973c85167b0e972dcdd568813a6953 with result from task 768 parsl.dataflow.dflow DEBUG: Task 768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/8c38477a-702b-4446-8aaf-34adc11e640f_isrForCrosstalkSources_1125923_54.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/8c38477a-702b-4446-8aaf-34adc11e640f_isrForCrosstalkSources_1125923_54.stderr parsl.dataflow.dflow INFO: Task 772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb512bed757b42a7ea6346988df08960 with result from task 772 parsl.dataflow.dflow DEBUG: Task 772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/581809aa-6fa0-4c7e-97c2-6d7d7a4e0266_isrForCrosstalkSources_1125903_24.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/581809aa-6fa0-4c7e-97c2-6d7d7a4e0266_isrForCrosstalkSources_1125903_24.stderr parsl.dataflow.dflow INFO: Task 774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27d38b1c01fb038dd1c6f38c79236d26 with result from task 774 parsl.dataflow.dflow DEBUG: Task 774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/ef1459ce-86da-4d42-9095-907424e3dc31_isrForCrosstalkSources_1125928_20.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/ef1459ce-86da-4d42-9095-907424e3dc31_isrForCrosstalkSources_1125928_20.stderr parsl.dataflow.dflow INFO: Task 756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7638f71a69e956b98efe6ca699731ea with result from task 756 parsl.dataflow.dflow DEBUG: Task 756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/9d470e4e-af5b-49d6-b982-6c32f290376c_isrForCrosstalkSources_1125944_59.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/9d470e4e-af5b-49d6-b982-6c32f290376c_isrForCrosstalkSources_1125944_59.stderr parsl.dataflow.dflow INFO: Task 773 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07e2ea4f13d530b0288c571dc5c39a9b with result from task 773 parsl.dataflow.dflow DEBUG: Task 773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/319b2458-8a83-4802-9276-714335542650_isrForCrosstalkSources_1125905_57.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/319b2458-8a83-4802-9276-714335542650_isrForCrosstalkSources_1125905_57.stderr parsl.dataflow.dflow INFO: Task 771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d0b72fc8e53654b89d6057bf04290b5 with result from task 771 parsl.dataflow.dflow DEBUG: Task 771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/ebc09346-f981-453a-a9e8-2570f8719c5e_isrForCrosstalkSources_1125893_55.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/ebc09346-f981-453a-a9e8-2570f8719c5e_isrForCrosstalkSources_1125893_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:14 proc_lsst.multi:146 [INFO] found job 16896664 in provider checkpoint proc_lsst.multi INFO: found job 16896664 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27028a6b6dd652dc940a7e807037bb3e with result from task 753 parsl.dataflow.dflow DEBUG: Task 753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d87e8e02-e2c9-4d91-b4b3-4cb40178597f_isrForCrosstalkSources_1125940_7.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d87e8e02-e2c9-4d91-b4b3-4cb40178597f_isrForCrosstalkSources_1125940_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:15 proc_lsst.multi:146 [INFO] found job 16896665 in provider checkpoint proc_lsst.multi INFO: found job 16896665 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6a67a854e697127dd67d4a94f6d7ea6 with result from task 748 parsl.dataflow.dflow DEBUG: Task 748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/c45aae5a-1318-4cb2-8e87-7b0b00c1806d_isrForCrosstalkSources_1125892_38.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/c45aae5a-1318-4cb2-8e87-7b0b00c1806d_isrForCrosstalkSources_1125892_38.stderr parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d4473c50cbec2cb36fac0798b99e6e7 with result from task 732 parsl.dataflow.dflow DEBUG: Task 732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/d9007c50-b50b-4541-8c2c-044285a99bce_isrForCrosstalkSources_1125914_43.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/d9007c50-b50b-4541-8c2c-044285a99bce_isrForCrosstalkSources_1125914_43.stderr parsl.dataflow.dflow INFO: Task 779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f51b208ed41ac4034d22912d9ab5fb59 with result from task 779 parsl.dataflow.dflow DEBUG: Task 779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1e2f2251-29d1-4f04-a944-f785977e6e5e_isrForCrosstalkSources_1125900_33.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1e2f2251-29d1-4f04-a944-f785977e6e5e_isrForCrosstalkSources_1125900_33.stderr parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbde892750711bf53d5c054d8f78f875 with result from task 730 parsl.dataflow.dflow DEBUG: Task 730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/1cce7af8-8011-498f-abc3-f8e97ab57984_isrForCrosstalkSources_1125897_44.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/1cce7af8-8011-498f-abc3-f8e97ab57984_isrForCrosstalkSources_1125897_44.stderr parsl.dataflow.dflow INFO: Task 775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 495dff810df904fcf8adf42e421f1eb2 with result from task 775 parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:15 proc_lsst.multi:146 [INFO] found job 16896666 in provider checkpoint proc_lsst.multi INFO: found job 16896666 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/1e28458e-4ee9-449f-9a15-a7dff646017e_isrForCrosstalkSources_1125899_49.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/1e28458e-4ee9-449f-9a15-a7dff646017e_isrForCrosstalkSources_1125899_49.stderr parsl.dataflow.dflow INFO: Task 757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4211dd26733daca9e031b9d40aca8f6 with result from task 757 parsl.dataflow.dflow DEBUG: Task 757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/a34518d4-0031-45d7-902d-f93ad2a2776a_isrForCrosstalkSources_1125927_48.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/a34518d4-0031-45d7-902d-f93ad2a2776a_isrForCrosstalkSources_1125927_48.stderr parsl.dataflow.dflow INFO: Task 776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23b827bb10dcba09414e3bb85388b3bc with result from task 776 parsl.dataflow.dflow DEBUG: Task 776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/d449ea95-61fd-49e1-9096-0b6899d0b744_isrForCrosstalkSources_1125895_25.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/d449ea95-61fd-49e1-9096-0b6899d0b744_isrForCrosstalkSources_1125895_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:16 proc_lsst.multi:146 [INFO] found job 16896667 in provider checkpoint proc_lsst.multi INFO: found job 16896667 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 080f91460f9be9ed46c4a9674bd7e707 with result from task 777 parsl.dataflow.dflow DEBUG: Task 777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/6549b3c9-195b-43ae-9e06-4e20125ea8f7_isrForCrosstalkSources_1125924_24.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/6549b3c9-195b-43ae-9e06-4e20125ea8f7_isrForCrosstalkSources_1125924_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:16 proc_lsst.multi:146 [INFO] found job 16896668 in provider checkpoint proc_lsst.multi INFO: found job 16896668 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:17 proc_lsst.multi:146 [INFO] found job 16896669 in provider checkpoint proc_lsst.multi INFO: found job 16896669 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:17 proc_lsst.multi:146 [INFO] found job 16896670 in provider checkpoint proc_lsst.multi INFO: found job 16896670 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:18 proc_lsst.multi:146 [INFO] found job 16896671 in provider checkpoint proc_lsst.multi INFO: found job 16896671 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2b022324d9e1e82933f2ad53445f2bc with result from task 780 parsl.dataflow.dflow DEBUG: Task 780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/23cc97d7-c41e-428b-852e-32fdaddff229_isrForCrosstalkSources_1125888_44.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/23cc97d7-c41e-428b-852e-32fdaddff229_isrForCrosstalkSources_1125888_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:18 proc_lsst.multi:146 [INFO] found job 16896672 in provider checkpoint proc_lsst.multi INFO: found job 16896672 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 811cbf4f4ab4c370b23e72c8cccb9a49 with result from task 781 parsl.dataflow.dflow DEBUG: Task 781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/2b03546e-8e57-4897-92c3-443227be21cb_isrForCrosstalkSources_1125938_41.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/2b03546e-8e57-4897-92c3-443227be21cb_isrForCrosstalkSources_1125938_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:19 proc_lsst.multi:146 [INFO] found job 16896673 in provider checkpoint proc_lsst.multi INFO: found job 16896673 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae0f891351ea1febb0ff14836f949f66 with result from task 778 parsl.dataflow.dflow DEBUG: Task 778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/85f2d6da-a658-43e6-bb8b-69091faf4731_isrForCrosstalkSources_1125942_3.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/85f2d6da-a658-43e6-bb8b-69091faf4731_isrForCrosstalkSources_1125942_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:19 proc_lsst.multi:146 [INFO] found job 16896674 in provider checkpoint proc_lsst.multi INFO: found job 16896674 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:20 proc_lsst.multi:146 [INFO] found job 16896675 in provider checkpoint proc_lsst.multi INFO: found job 16896675 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:20 proc_lsst.multi:146 [INFO] found job 16896676 in provider checkpoint proc_lsst.multi INFO: found job 16896676 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc7e912dea2fd65906affb978df42437 with result from task 784 parsl.dataflow.dflow DEBUG: Task 784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4cba92f3-b216-49b8-be84-fc391c3758bd_isrForCrosstalkSources_1125893_27.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4cba92f3-b216-49b8-be84-fc391c3758bd_isrForCrosstalkSources_1125893_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:21 proc_lsst.multi:146 [INFO] found job 16896678 in provider checkpoint proc_lsst.multi INFO: found job 16896678 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5ab132ab754afaa45614d6fa70ebb25 with result from task 789 parsl.dataflow.dflow DEBUG: Task 789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/61ed8451-e7db-4c08-a466-ef985182e642_isrForCrosstalkSources_1125932_4.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/61ed8451-e7db-4c08-a466-ef985182e642_isrForCrosstalkSources_1125932_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:21 proc_lsst.multi:146 [INFO] found job 16896679 in provider checkpoint proc_lsst.multi INFO: found job 16896679 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fed8530ed3c9ef0b0a80d1d8b4a697f with result from task 788 parsl.dataflow.dflow DEBUG: Task 788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/d57e8ffc-502a-4533-97a3-414dbba8dcca_isrForCrosstalkSources_1125892_11.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/d57e8ffc-502a-4533-97a3-414dbba8dcca_isrForCrosstalkSources_1125892_11.stderr parsl.dataflow.dflow INFO: Task 783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 920e4902f93e1c0ea02fe1c3193c8229 with result from task 783 parsl.dataflow.dflow DEBUG: Task 783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/52c7774f-424f-462b-8a17-b342d9d3f525_isrForCrosstalkSources_1125920_1.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/52c7774f-424f-462b-8a17-b342d9d3f525_isrForCrosstalkSources_1125920_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:22 proc_lsst.multi:146 [INFO] found job 16896680 in provider checkpoint proc_lsst.multi INFO: found job 16896680 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3047a1a7549962f027d497b4e8a9bb55 with result from task 790 parsl.dataflow.dflow DEBUG: Task 790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/77d92217-e7ac-4fc3-a8bf-8bae61360866_isrForCrosstalkSources_1125934_43.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/77d92217-e7ac-4fc3-a8bf-8bae61360866_isrForCrosstalkSources_1125934_43.stderr parsl.dataflow.dflow INFO: Task 797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 301041647043c40f2939a0027f13a8ea with result from task 797 parsl.dataflow.dflow DEBUG: Task 797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/825162ce-075f-4c34-bdbc-30e6c0b224f2_isrForCrosstalkSources_1125941_32.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/825162ce-075f-4c34-bdbc-30e6c0b224f2_isrForCrosstalkSources_1125941_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:22 proc_lsst.multi:146 [INFO] found job 16896681 in provider checkpoint proc_lsst.multi INFO: found job 16896681 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddb85a378ba3869dba317f054ccc1283 with result from task 801 parsl.dataflow.dflow DEBUG: Task 801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/c5e2dae4-f0b3-436b-a61f-da8fabf02b31_isrForCrosstalkSources_1125946_56.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/c5e2dae4-f0b3-436b-a61f-da8fabf02b31_isrForCrosstalkSources_1125946_56.stderr parsl.dataflow.dflow INFO: Task 795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a7d00c4980406c9fb98f75fc34443b7 with result from task 795 parsl.dataflow.dflow DEBUG: Task 795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/773e047f-ca47-412e-871c-492d7f06dd1e_isrForCrosstalkSources_1125904_60.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/773e047f-ca47-412e-871c-492d7f06dd1e_isrForCrosstalkSources_1125904_60.stderr parsl.dataflow.dflow INFO: Task 793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0b1dc2feb1f8d82e9f0bd79011c4f14 with result from task 793 parsl.dataflow.dflow DEBUG: Task 793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/79a07bba-4a6c-449a-9c06-bb27ba63c1bb_isrForCrosstalkSources_1125924_58.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/79a07bba-4a6c-449a-9c06-bb27ba63c1bb_isrForCrosstalkSources_1125924_58.stderr parsl.dataflow.dflow INFO: Task 823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a5a612325ce4964881f92126d0c7e36 with result from task 823 parsl.dataflow.dflow DEBUG: Task 823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/2d67f7e7-919a-448e-beb2-b5875202a26b_isrForCrosstalkSources_1125901_27.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/2d67f7e7-919a-448e-beb2-b5875202a26b_isrForCrosstalkSources_1125901_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:23 proc_lsst.multi:146 [INFO] found job 16896682 in provider checkpoint proc_lsst.multi INFO: found job 16896682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08a299091c867ad5fbe76e22550eb4fc with result from task 811 parsl.dataflow.dflow DEBUG: Task 811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3a06442a-3b8e-4bf5-b1c5-7ca4d24f2711_isrForCrosstalkSources_1125912_31.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3a06442a-3b8e-4bf5-b1c5-7ca4d24f2711_isrForCrosstalkSources_1125912_31.stderr parsl.dataflow.dflow INFO: Task 800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 972ebd609efe0ed3a999401555f9da0c with result from task 800 parsl.dataflow.dflow DEBUG: Task 800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/8d149064-3467-476c-9c6b-c882bc94eb36_isrForCrosstalkSources_1125912_47.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/8d149064-3467-476c-9c6b-c882bc94eb36_isrForCrosstalkSources_1125912_47.stderr parsl.dataflow.dflow INFO: Task 820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2daf8cefc64e9be2f7ab8e7fe7aa9840 with result from task 820 parsl.dataflow.dflow DEBUG: Task 820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/571c35f3-af38-449c-8aea-69b37b40e1e9_isrForCrosstalkSources_1125895_32.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/571c35f3-af38-449c-8aea-69b37b40e1e9_isrForCrosstalkSources_1125895_32.stderr parsl.dataflow.dflow INFO: Task 791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16727a4626554f4c892588ec7e2624e4 with result from task 791 parsl.dataflow.dflow DEBUG: Task 791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/d4bc819f-861c-4ab4-a68e-49db0daf70cd_isrForCrosstalkSources_1125926_33.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/d4bc819f-861c-4ab4-a68e-49db0daf70cd_isrForCrosstalkSources_1125926_33.stderr parsl.dataflow.dflow INFO: Task 786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c93efa40a39ca7f7cd468a647c41ff7 with result from task 786 parsl.dataflow.dflow DEBUG: Task 786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/412f19b0-36aa-474a-8d8b-4ee0e8b02c35_isrForCrosstalkSources_1125917_40.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/412f19b0-36aa-474a-8d8b-4ee0e8b02c35_isrForCrosstalkSources_1125917_40.stderr parsl.dataflow.dflow INFO: Task 792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4a0ca3550d96ed477e05ad59027117b with result from task 792 parsl.dataflow.dflow DEBUG: Task 792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/0a38e9a0-8f5f-4ea2-91b8-396117077190_isrForCrosstalkSources_1125909_27.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/0a38e9a0-8f5f-4ea2-91b8-396117077190_isrForCrosstalkSources_1125909_27.stderr parsl.dataflow.dflow INFO: Task 785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33bccc1dd4a7f4f5cdfccdb162a7feda with result from task 785 parsl.dataflow.dflow DEBUG: Task 785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/2534042c-26ed-4321-bbc4-9cf9cb3d8d81_isrForCrosstalkSources_1125918_30.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/2534042c-26ed-4321-bbc4-9cf9cb3d8d81_isrForCrosstalkSources_1125918_30.stderr parsl.dataflow.dflow INFO: Task 802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1659f5473aa9799b65eff96202c0eb1a with result from task 802 parsl.dataflow.dflow DEBUG: Task 802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d82c489e-3205-4ae7-b92e-1c2f6ce14f4c_isrForCrosstalkSources_1125934_30.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d82c489e-3205-4ae7-b92e-1c2f6ce14f4c_isrForCrosstalkSources_1125934_30.stderr parsl.dataflow.dflow INFO: Task 817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8de203b7a4ae43db051928711c5551c with result from task 817 parsl.dataflow.dflow DEBUG: Task 817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/d402793e-3a8d-4799-9a7d-91b55bed49df_isrForCrosstalkSources_1125901_13.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/d402793e-3a8d-4799-9a7d-91b55bed49df_isrForCrosstalkSources_1125901_13.stderr parsl.dataflow.dflow INFO: Task 799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b47ec116e1a4a362138bed0485c5d569 with result from task 799 parsl.dataflow.dflow DEBUG: Task 799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/7f7142b2-aebb-4534-9336-277b00fe3cf7_isrForCrosstalkSources_1125900_23.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/7f7142b2-aebb-4534-9336-277b00fe3cf7_isrForCrosstalkSources_1125900_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:23 proc_lsst.multi:146 [INFO] found job 16896683 in provider checkpoint proc_lsst.multi INFO: found job 16896683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cea686959ab58bf83748a0c89c00a82e with result from task 816 parsl.dataflow.dflow DEBUG: Task 816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/c12b7b77-c92c-4683-ac3c-82f63beff290_isrForCrosstalkSources_1125911_10.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/c12b7b77-c92c-4683-ac3c-82f63beff290_isrForCrosstalkSources_1125911_10.stderr parsl.dataflow.dflow INFO: Task 808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cb89e4e450669846a8c1a31a0ccd529 with result from task 808 parsl.dataflow.dflow DEBUG: Task 808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/3a94667e-48bf-4577-ab7e-74c00bf41afd_isrForCrosstalkSources_1125894_9.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/3a94667e-48bf-4577-ab7e-74c00bf41afd_isrForCrosstalkSources_1125894_9.stderr parsl.dataflow.dflow INFO: Task 782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 816a0bf72e8c8d5113f38784d42d7486 with result from task 782 parsl.dataflow.dflow DEBUG: Task 782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/7ecee4f1-393e-4218-a2b4-074fe02535ae_isrForCrosstalkSources_1125937_32.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/7ecee4f1-393e-4218-a2b4-074fe02535ae_isrForCrosstalkSources_1125937_32.stderr parsl.dataflow.dflow INFO: Task 798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d23fe99f20855c1024ffe3cce74be5b with result from task 798 parsl.dataflow.dflow DEBUG: Task 798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/a3d8521a-aa0d-4ae6-b4e0-b5b805e9cf5c_isrForCrosstalkSources_1125924_7.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/a3d8521a-aa0d-4ae6-b4e0-b5b805e9cf5c_isrForCrosstalkSources_1125924_7.stderr parsl.dataflow.dflow INFO: Task 794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e3951d7dad97b8145a4565cd7df9a72 with result from task 794 parsl.dataflow.dflow DEBUG: Task 794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f4729d3a-cfc0-4c54-a820-3371cbca83cc_isrForCrosstalkSources_1125894_40.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f4729d3a-cfc0-4c54-a820-3371cbca83cc_isrForCrosstalkSources_1125894_40.stderr parsl.dataflow.dflow INFO: Task 813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 343314ae6ca9a31831f3d0bbd0a10dc7 with result from task 813 parsl.dataflow.dflow DEBUG: Task 813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/190aee37-d5fe-4de4-a860-413ccbf617a6_isrForCrosstalkSources_1125937_25.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/190aee37-d5fe-4de4-a860-413ccbf617a6_isrForCrosstalkSources_1125937_25.stderr parsl.dataflow.dflow INFO: Task 824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0b9be01fa400be6212cf44575d10964 with result from task 824 parsl.dataflow.dflow DEBUG: Task 824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/d6c99096-fdf8-4576-9418-b8a307267b04_isrForCrosstalkSources_1125916_51.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/d6c99096-fdf8-4576-9418-b8a307267b04_isrForCrosstalkSources_1125916_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:24 proc_lsst.multi:146 [INFO] found job 16896684 in provider checkpoint proc_lsst.multi INFO: found job 16896684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0b3c18d431c6fc15780fc25eef08cca with result from task 822 parsl.dataflow.dflow DEBUG: Task 822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/821319cb-fbfb-4465-aa42-48667d05a398_isrForCrosstalkSources_1125908_27.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/821319cb-fbfb-4465-aa42-48667d05a398_isrForCrosstalkSources_1125908_27.stderr parsl.dataflow.dflow INFO: Task 809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 308437c320d1f47b35d80ae45315c458 with result from task 809 parsl.dataflow.dflow DEBUG: Task 809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/9fd960f9-2c4e-488d-8a3e-36e3c45d2236_isrForCrosstalkSources_1125905_4.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/9fd960f9-2c4e-488d-8a3e-36e3c45d2236_isrForCrosstalkSources_1125905_4.stderr parsl.dataflow.dflow INFO: Task 812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56f4f20ec6068168ed3327f66bed44d3 with result from task 812 parsl.dataflow.dflow DEBUG: Task 812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/e94fa875-7008-4732-aa41-c1a2fbd6ccfe_isrForCrosstalkSources_1125936_1.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/e94fa875-7008-4732-aa41-c1a2fbd6ccfe_isrForCrosstalkSources_1125936_1.stderr parsl.dataflow.dflow INFO: Task 814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac5951f85c10dacff0a6a1bf6643470e with result from task 814 parsl.dataflow.dflow DEBUG: Task 814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/64ae782f-c567-42cb-b876-5e6cee337063_isrForCrosstalkSources_1125888_36.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/64ae782f-c567-42cb-b876-5e6cee337063_isrForCrosstalkSources_1125888_36.stderr parsl.dataflow.dflow INFO: Task 803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fe40f235aefb1d3aba13b27b6c3f3c8 with result from task 803 parsl.dataflow.dflow DEBUG: Task 803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/92646f71-0bbf-49b9-8d71-4be5b9b618d0_isrForCrosstalkSources_1125932_17.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/92646f71-0bbf-49b9-8d71-4be5b9b618d0_isrForCrosstalkSources_1125932_17.stderr parsl.dataflow.dflow INFO: Task 821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4968bfc9049bc91f58dbcceff815da46 with result from task 821 parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:24 proc_lsst.multi:146 [INFO] found job 16896685 in provider checkpoint proc_lsst.multi INFO: found job 16896685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e46dbcf7-a3d7-4aa1-95ff-72b305d1919d_isrForCrosstalkSources_1125937_53.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e46dbcf7-a3d7-4aa1-95ff-72b305d1919d_isrForCrosstalkSources_1125937_53.stderr parsl.dataflow.dflow INFO: Task 810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cc6caad8c5229a97101a8baf358a643 with result from task 810 parsl.dataflow.dflow DEBUG: Task 810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ef7813e9-bcc4-470f-83fb-b5d04326ce12_isrForCrosstalkSources_1125894_7.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ef7813e9-bcc4-470f-83fb-b5d04326ce12_isrForCrosstalkSources_1125894_7.stderr parsl.dataflow.dflow INFO: Task 807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f7eca7ae2da22e2207de24b819f955d with result from task 807 parsl.dataflow.dflow DEBUG: Task 807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/16bc2415-ae65-4bd6-acbb-93d1f485c80c_isrForCrosstalkSources_1125946_10.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/16bc2415-ae65-4bd6-acbb-93d1f485c80c_isrForCrosstalkSources_1125946_10.stderr parsl.dataflow.dflow INFO: Task 825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf44a789fb316cc5ec510209c407933c with result from task 825 parsl.dataflow.dflow DEBUG: Task 825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/3885782b-2de3-42cd-808f-ccfd58185076_isrForCrosstalkSources_1125918_40.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/3885782b-2de3-42cd-808f-ccfd58185076_isrForCrosstalkSources_1125918_40.stderr parsl.dataflow.dflow INFO: Task 805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 306e2761f99aa3a04e3ad61fc2170281 with result from task 805 parsl.dataflow.dflow DEBUG: Task 805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/2657dc2a-de4d-44e1-a46b-d9cc7625dfaf_isrForCrosstalkSources_1125905_3.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/2657dc2a-de4d-44e1-a46b-d9cc7625dfaf_isrForCrosstalkSources_1125905_3.stderr parsl.dataflow.dflow INFO: Task 819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af2e34c4840f9ade07574e23cc1fef31 with result from task 819 parsl.dataflow.dflow DEBUG: Task 819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/10166b6a-856a-4fe0-9e36-68fb54f8e7da_isrForCrosstalkSources_1125890_9.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/10166b6a-856a-4fe0-9e36-68fb54f8e7da_isrForCrosstalkSources_1125890_9.stderr parsl.dataflow.dflow INFO: Task 818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e57fa057a505038357b76074321c5c8 with result from task 818 parsl.dataflow.dflow DEBUG: Task 818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/b6daafd2-0149-4d61-a62d-9ed1c18ebba7_isrForCrosstalkSources_1125902_42.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/b6daafd2-0149-4d61-a62d-9ed1c18ebba7_isrForCrosstalkSources_1125902_42.stderr parsl.dataflow.dflow INFO: Task 815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7989c962b275bbcd26b331f08001c8a with result from task 815 parsl.dataflow.dflow DEBUG: Task 815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/2898608d-51a1-4fc8-9243-56a9c6b4e88b_isrForCrosstalkSources_1125918_58.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/2898608d-51a1-4fc8-9243-56a9c6b4e88b_isrForCrosstalkSources_1125918_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:25 proc_lsst.multi:146 [INFO] found job 16896686 in provider checkpoint proc_lsst.multi INFO: found job 16896686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33f6cd1781404eef1b0f04254aab3add with result from task 804 parsl.dataflow.dflow DEBUG: Task 804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/86b730fe-bc65-4a6c-bcf9-6a81a1637339_isrForCrosstalkSources_1125915_37.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/86b730fe-bc65-4a6c-bcf9-6a81a1637339_isrForCrosstalkSources_1125915_37.stderr parsl.dataflow.dflow INFO: Task 806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a46978b9cb3c6b1f9a9da06522d7805e with result from task 806 parsl.dataflow.dflow DEBUG: Task 806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/bbc48eed-084b-4244-be11-3d7fec17b472_isrForCrosstalkSources_1125912_28.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/bbc48eed-084b-4244-be11-3d7fec17b472_isrForCrosstalkSources_1125912_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:25 proc_lsst.multi:146 [INFO] found job 16896687 in provider checkpoint proc_lsst.multi INFO: found job 16896687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0d1dfcacb4f4ee520a4c03f6eb9ca7f with result from task 796 parsl.dataflow.dflow DEBUG: Task 796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/83bd98f5-eefd-4959-9de6-d969047d5bc9_isrForCrosstalkSources_1125944_52.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/83bd98f5-eefd-4959-9de6-d969047d5bc9_isrForCrosstalkSources_1125944_52.stderr parsl.dataflow.dflow INFO: Task 828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a345dbd31f3df60711483d2b4b38dc3 with result from task 828 parsl.dataflow.dflow DEBUG: Task 828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/b674b963-912b-4198-9651-a1f63c941a03_isrForCrosstalkSources_1125944_16.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/b674b963-912b-4198-9651-a1f63c941a03_isrForCrosstalkSources_1125944_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:26 proc_lsst.multi:146 [INFO] found job 16896688 in provider checkpoint proc_lsst.multi INFO: found job 16896688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19abdfb9d42501c6a990e3095500bff4 with result from task 787 parsl.dataflow.dflow DEBUG: Task 787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/b7468187-cc4b-43da-8f7d-7545acf42797_isrForCrosstalkSources_1125917_53.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/b7468187-cc4b-43da-8f7d-7545acf42797_isrForCrosstalkSources_1125917_53.stderr parsl.dataflow.dflow INFO: Task 829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab714c8e3fd82370de2a9aaa3ece0204 with result from task 829 parsl.dataflow.dflow DEBUG: Task 829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/db8ceb9a-6375-423d-8873-0bf4003ef4d5_isrForCrosstalkSources_1125924_27.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/db8ceb9a-6375-423d-8873-0bf4003ef4d5_isrForCrosstalkSources_1125924_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:26 proc_lsst.multi:146 [INFO] found job 16896689 in provider checkpoint proc_lsst.multi INFO: found job 16896689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17bc8532ef4a407b15b8f4fc967d91aa with result from task 826 parsl.dataflow.dflow DEBUG: Task 826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/12636828-5316-48b5-b4c9-24f1371e413e_isrForCrosstalkSources_1125889_39.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/12636828-5316-48b5-b4c9-24f1371e413e_isrForCrosstalkSources_1125889_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:27 proc_lsst.multi:146 [INFO] found job 16896690 in provider checkpoint proc_lsst.multi INFO: found job 16896690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c12c2648e87f236106a17c6e3b85922 with result from task 830 parsl.dataflow.dflow DEBUG: Task 830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/f76b9767-7209-4ab9-b34b-98aed6fd50ed_isrForCrosstalkSources_1125924_29.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/f76b9767-7209-4ab9-b34b-98aed6fd50ed_isrForCrosstalkSources_1125924_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:27 proc_lsst.multi:146 [INFO] found job 16896691 in provider checkpoint proc_lsst.multi INFO: found job 16896691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb8472abfc7757e5cf6d93a52a3f52c7 with result from task 831 parsl.dataflow.dflow DEBUG: Task 831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/e94ee1d3-988c-42b5-b821-bfed04429114_isrForCrosstalkSources_1125914_37.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/e94ee1d3-988c-42b5-b821-bfed04429114_isrForCrosstalkSources_1125914_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:28 proc_lsst.multi:146 [INFO] found job 16896692 in provider checkpoint proc_lsst.multi INFO: found job 16896692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 972f29402c8ae880735b6627af2f5208 with result from task 827 parsl.dataflow.dflow DEBUG: Task 827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/67110e27-ea35-416c-83ad-9a1f6c2dfb77_isrForCrosstalkSources_1125925_51.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/67110e27-ea35-416c-83ad-9a1f6c2dfb77_isrForCrosstalkSources_1125925_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:28 proc_lsst.multi:146 [INFO] found job 16896693 in provider checkpoint proc_lsst.multi INFO: found job 16896693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:29 proc_lsst.multi:146 [INFO] found job 16896694 in provider checkpoint proc_lsst.multi INFO: found job 16896694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:29 proc_lsst.multi:146 [INFO] found job 16896695 in provider checkpoint proc_lsst.multi INFO: found job 16896695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:30 proc_lsst.multi:146 [INFO] found job 16896696 in provider checkpoint proc_lsst.multi INFO: found job 16896696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:30 proc_lsst.multi:146 [INFO] found job 16896697 in provider checkpoint proc_lsst.multi INFO: found job 16896697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:31 proc_lsst.multi:146 [INFO] found job 16896698 in provider checkpoint proc_lsst.multi INFO: found job 16896698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:31 proc_lsst.multi:146 [INFO] found job 16896699 in provider checkpoint proc_lsst.multi INFO: found job 16896699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:31 proc_lsst.multi:146 [INFO] found job 16896700 in provider checkpoint proc_lsst.multi INFO: found job 16896700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:31 proc_lsst.multi:146 [INFO] found job 16896701 in provider checkpoint proc_lsst.multi INFO: found job 16896701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3fed2711830e333c85f3fb5b8039536 with result from task 839 parsl.dataflow.dflow DEBUG: Task 839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e9efc920-6bd0-475c-96c1-5aa43e25ebf1_isrForCrosstalkSources_1125901_9.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e9efc920-6bd0-475c-96c1-5aa43e25ebf1_isrForCrosstalkSources_1125901_9.stderr parsl.dataflow.dflow INFO: Task 833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffed6053a5c07f3f41a566ff7f97deb7 with result from task 833 parsl.dataflow.dflow DEBUG: Task 833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/7eb08dfd-e7d5-45c1-837a-0cff078bd8dd_isrForCrosstalkSources_1125899_14.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/7eb08dfd-e7d5-45c1-837a-0cff078bd8dd_isrForCrosstalkSources_1125899_14.stderr parsl.dataflow.dflow INFO: Task 832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dbd00db6f8414773fd01a540f71d108 with result from task 832 parsl.dataflow.dflow DEBUG: Task 832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/520d3928-8750-4307-8c07-6752e3516eaf_isrForCrosstalkSources_1125931_54.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/520d3928-8750-4307-8c07-6752e3516eaf_isrForCrosstalkSources_1125931_54.stderr parsl.dataflow.dflow INFO: Task 834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca88877b138d330535b0928d82faefb9 with result from task 834 parsl.dataflow.dflow DEBUG: Task 834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/18fbfb6e-88b7-4f4b-bdbb-3d9554862ca4_isrForCrosstalkSources_1125928_40.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/18fbfb6e-88b7-4f4b-bdbb-3d9554862ca4_isrForCrosstalkSources_1125928_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:32 proc_lsst.multi:146 [INFO] found job 16896702 in provider checkpoint proc_lsst.multi INFO: found job 16896702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e29abe1906353d9256f306a983504e68 with result from task 845 parsl.dataflow.dflow DEBUG: Task 845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/aed4bd26-fe85-49a9-b562-b56fe7fa19a8_isrForCrosstalkSources_1125895_56.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/aed4bd26-fe85-49a9-b562-b56fe7fa19a8_isrForCrosstalkSources_1125895_56.stderr parsl.dataflow.dflow INFO: Task 836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be31f5bbd65b5ab5f1c16d6a585a25d7 with result from task 836 parsl.dataflow.dflow DEBUG: Task 836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/9399e372-5168-491b-960b-167310281e07_isrForCrosstalkSources_1125937_5.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/9399e372-5168-491b-960b-167310281e07_isrForCrosstalkSources_1125937_5.stderr parsl.dataflow.dflow INFO: Task 851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb76d66c8d65a705958548998469aba7 with result from task 851 parsl.dataflow.dflow DEBUG: Task 851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/b612b176-3293-421e-91a5-9d0ed34f5607_isrForCrosstalkSources_1125921_27.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/b612b176-3293-421e-91a5-9d0ed34f5607_isrForCrosstalkSources_1125921_27.stderr parsl.dataflow.dflow INFO: Task 835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf72c186dca04c5179bf6a4f0faf2c9b with result from task 835 parsl.dataflow.dflow DEBUG: Task 835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/b50d5fc8-6146-4dd3-8144-4a1b1b11383f_isrForCrosstalkSources_1125945_52.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/b50d5fc8-6146-4dd3-8144-4a1b1b11383f_isrForCrosstalkSources_1125945_52.stderr parsl.dataflow.dflow INFO: Task 843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d1e09d245f48598e1b0b71d0861fb00 with result from task 843 parsl.dataflow.dflow DEBUG: Task 843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/a0a609cf-d137-4ede-810a-236e06f1cff9_isrForCrosstalkSources_1125895_53.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/a0a609cf-d137-4ede-810a-236e06f1cff9_isrForCrosstalkSources_1125895_53.stderr parsl.dataflow.dflow INFO: Task 837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 276469cf98d9a07a7ed68076d7e2c39d with result from task 837 parsl.dataflow.dflow DEBUG: Task 837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/7a02a875-4fcb-497b-9ec0-558708e9fb11_isrForCrosstalkSources_1125899_23.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/7a02a875-4fcb-497b-9ec0-558708e9fb11_isrForCrosstalkSources_1125899_23.stderr parsl.dataflow.dflow INFO: Task 838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c16147833c378c5fdcdceb4abbb73a77 with result from task 838 parsl.dataflow.dflow DEBUG: Task 838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/81299053-63fe-4e7f-aebf-4ec27ec112f1_isrForCrosstalkSources_1125937_7.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/81299053-63fe-4e7f-aebf-4ec27ec112f1_isrForCrosstalkSources_1125937_7.stderr parsl.dataflow.dflow INFO: Task 844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7bf9df968831efa31a72af4b7d676e9 with result from task 844 parsl.dataflow.dflow DEBUG: Task 844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/dfde5563-5524-484d-b214-12bea7b7e475_isrForCrosstalkSources_1125931_21.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/dfde5563-5524-484d-b214-12bea7b7e475_isrForCrosstalkSources_1125931_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:33 proc_lsst.multi:146 [INFO] found job 16896703 in provider checkpoint proc_lsst.multi INFO: found job 16896703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fff5fade69dc0df6f301df5973d1d1cc with result from task 860 parsl.dataflow.dflow DEBUG: Task 860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/9b287236-5f02-4394-9a62-c11bb5f1ca6b_isrForCrosstalkSources_1125933_34.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/9b287236-5f02-4394-9a62-c11bb5f1ca6b_isrForCrosstalkSources_1125933_34.stderr parsl.dataflow.dflow INFO: Task 859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7aab0e1aff4f95650ad2644b45da836 with result from task 859 parsl.dataflow.dflow DEBUG: Task 859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/794bdd86-ace1-4af2-bfff-95a63e7fcf81_isrForCrosstalkSources_1125926_21.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/794bdd86-ace1-4af2-bfff-95a63e7fcf81_isrForCrosstalkSources_1125926_21.stderr parsl.dataflow.dflow INFO: Task 840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4530296900bca00359185b35155ee97a with result from task 840 parsl.dataflow.dflow DEBUG: Task 840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/10545506-5212-41e5-b45b-9ccb887877df_isrForCrosstalkSources_1125899_51.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/10545506-5212-41e5-b45b-9ccb887877df_isrForCrosstalkSources_1125899_51.stderr parsl.dataflow.dflow INFO: Task 853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 593f3eff09c073d63c31b043f66d2942 with result from task 853 parsl.dataflow.dflow DEBUG: Task 853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/3e1b85cf-188d-4687-a077-ac28430a12dc_isrForCrosstalkSources_1125894_2.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/3e1b85cf-188d-4687-a077-ac28430a12dc_isrForCrosstalkSources_1125894_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:33 proc_lsst.multi:146 [INFO] found job 16896704 in provider checkpoint proc_lsst.multi INFO: found job 16896704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72b490cd63c2d8461c1d41b6ed0ba470 with result from task 863 parsl.dataflow.dflow DEBUG: Task 863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2ea8adda-32e4-465c-b763-20c89ee4f44f_isrForCrosstalkSources_1125898_11.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2ea8adda-32e4-465c-b763-20c89ee4f44f_isrForCrosstalkSources_1125898_11.stderr parsl.dataflow.dflow INFO: Task 842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0e0bdd60e3075bf53530dab28aa085b with result from task 842 parsl.dataflow.dflow DEBUG: Task 842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/387174f1-51ab-4ad2-97dc-4bad64fd7072_isrForCrosstalkSources_1125895_7.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/387174f1-51ab-4ad2-97dc-4bad64fd7072_isrForCrosstalkSources_1125895_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:34 proc_lsst.multi:146 [INFO] found job 16896705 in provider checkpoint proc_lsst.multi INFO: found job 16896705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f91c5dd00bf16b52bdc34830fa8013d8 with result from task 841 parsl.dataflow.dflow DEBUG: Task 841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/7e7d8a45-c860-4e29-8a42-cee3c770d5a8_isrForCrosstalkSources_1125938_30.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/7e7d8a45-c860-4e29-8a42-cee3c770d5a8_isrForCrosstalkSources_1125938_30.stderr parsl.dataflow.dflow INFO: Task 865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54b574dca1d501ade3e71b23ce791549 with result from task 865 parsl.dataflow.dflow DEBUG: Task 865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/4e35278b-4c4c-490c-b47f-856ecbc25ec9_isrForCrosstalkSources_1125918_54.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/4e35278b-4c4c-490c-b47f-856ecbc25ec9_isrForCrosstalkSources_1125918_54.stderr parsl.dataflow.dflow INFO: Task 875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23d615dfb45333c89bcea8a81090d1b0 with result from task 875 parsl.dataflow.dflow DEBUG: Task 875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d413550f-566d-40de-957e-2d26c23b002b_isrForCrosstalkSources_1125923_27.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d413550f-566d-40de-957e-2d26c23b002b_isrForCrosstalkSources_1125923_27.stderr parsl.dataflow.dflow INFO: Task 861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 556a995883f93d01b573be4bf845f550 with result from task 861 parsl.dataflow.dflow DEBUG: Task 861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/e2dc0513-2ce7-409a-8674-68123b05e22f_isrForCrosstalkSources_1125945_26.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/e2dc0513-2ce7-409a-8674-68123b05e22f_isrForCrosstalkSources_1125945_26.stderr parsl.dataflow.dflow INFO: Task 872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c98aad997f910e6912a7e202ef7caab with result from task 872 parsl.dataflow.dflow DEBUG: Task 872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/d6e9ca62-e041-4912-a4c2-8f9e798e3e84_isrForCrosstalkSources_1125893_40.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/d6e9ca62-e041-4912-a4c2-8f9e798e3e84_isrForCrosstalkSources_1125893_40.stderr parsl.dataflow.dflow INFO: Task 866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30eb076ebaa4bfba0eb265c3f71ac8ad with result from task 866 parsl.dataflow.dflow DEBUG: Task 866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/28567900-20eb-4ca2-a4b4-9ffd2a500225_isrForCrosstalkSources_1125903_12.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/28567900-20eb-4ca2-a4b4-9ffd2a500225_isrForCrosstalkSources_1125903_12.stderr parsl.dataflow.dflow INFO: Task 867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32b3f8129b7e80b3174f41428098e3ad with result from task 867 parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:34 proc_lsst.multi:146 [INFO] found job 16896706 in provider checkpoint proc_lsst.multi INFO: found job 16896706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/5bbf1971-ddc4-411d-a3c9-3a541851be42_isrForCrosstalkSources_1125943_27.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/5bbf1971-ddc4-411d-a3c9-3a541851be42_isrForCrosstalkSources_1125943_27.stderr parsl.dataflow.dflow INFO: Task 870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24cc835540c641bcdfea702eaafda024 with result from task 870 parsl.dataflow.dflow DEBUG: Task 870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/e7824198-358e-49a7-a524-0f66c35cb9d5_isrForCrosstalkSources_1125897_61.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/e7824198-358e-49a7-a524-0f66c35cb9d5_isrForCrosstalkSources_1125897_61.stderr parsl.dataflow.dflow INFO: Task 858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4edd7f87719231113514ec68dd1aad2 with result from task 858 parsl.dataflow.dflow DEBUG: Task 858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b16d9cfc-f0e6-4fe9-a8f7-2733bfd607ba_isrForCrosstalkSources_1125938_18.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b16d9cfc-f0e6-4fe9-a8f7-2733bfd607ba_isrForCrosstalkSources_1125938_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:35 proc_lsst.multi:146 [INFO] found job 16896707 in provider checkpoint proc_lsst.multi INFO: found job 16896707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d99ddeb6d9ce350b6c065cc8b18dd5d with result from task 877 parsl.dataflow.dflow DEBUG: Task 877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/af8480ad-4e23-4586-9258-22784d6ddaf7_isrForCrosstalkSources_1125947_22.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/af8480ad-4e23-4586-9258-22784d6ddaf7_isrForCrosstalkSources_1125947_22.stderr parsl.dataflow.dflow INFO: Task 869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d778c313bf29ce8294d50e0602513a18 with result from task 869 parsl.dataflow.dflow DEBUG: Task 869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/06f222bb-4413-4dbb-8410-11dae4dbf96e_isrForCrosstalkSources_1125908_17.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/06f222bb-4413-4dbb-8410-11dae4dbf96e_isrForCrosstalkSources_1125908_17.stderr parsl.dataflow.dflow INFO: Task 852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d741cd5392e1d6e1f674a86ba467b618 with result from task 852 parsl.dataflow.dflow DEBUG: Task 852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/f0f06a00-ff97-4676-a305-94bfccafb336_isrForCrosstalkSources_1125932_48.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/f0f06a00-ff97-4676-a305-94bfccafb336_isrForCrosstalkSources_1125932_48.stderr parsl.dataflow.dflow INFO: Task 868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1b9ac846468186ba53d793ce124e5f8 with result from task 868 parsl.dataflow.dflow DEBUG: Task 868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3c437f67-cb01-4a77-af19-4afb313b937d_isrForCrosstalkSources_1125941_42.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3c437f67-cb01-4a77-af19-4afb313b937d_isrForCrosstalkSources_1125941_42.stderr parsl.dataflow.dflow INFO: Task 881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a1dbd8d402c96eeed966f59dc21ce96 with result from task 881 parsl.dataflow.dflow DEBUG: Task 881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/5366fabf-3af4-43b0-b4c2-b62542cf89c7_isrForCrosstalkSources_1125927_8.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/5366fabf-3af4-43b0-b4c2-b62542cf89c7_isrForCrosstalkSources_1125927_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:35 proc_lsst.multi:146 [INFO] found job 16896708 in provider checkpoint proc_lsst.multi INFO: found job 16896708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29cd2ba174d4bb126c81e103ddc05835 with result from task 880 parsl.dataflow.dflow DEBUG: Task 880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/039187db-a8e4-4845-a7fd-f73b45054530_isrForCrosstalkSources_1125933_6.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/039187db-a8e4-4845-a7fd-f73b45054530_isrForCrosstalkSources_1125933_6.stderr parsl.dataflow.dflow INFO: Task 864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc1d795f5b51f17987a2d6e2fa9d5ad4 with result from task 864 parsl.dataflow.dflow DEBUG: Task 864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/a6aacb45-4839-420d-83a9-f8f1ceb336ac_isrForCrosstalkSources_1125928_52.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/a6aacb45-4839-420d-83a9-f8f1ceb336ac_isrForCrosstalkSources_1125928_52.stderr parsl.dataflow.dflow INFO: Task 862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7efdbaafb0bb1ae7df8414aeda7c2f22 with result from task 862 parsl.dataflow.dflow DEBUG: Task 862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/aeb293f6-44ca-4884-865b-d667d572a2ed_isrForCrosstalkSources_1125919_61.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/aeb293f6-44ca-4884-865b-d667d572a2ed_isrForCrosstalkSources_1125919_61.stderr parsl.dataflow.dflow INFO: Task 873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fd7dc5bff4289906b379f798f5da746 with result from task 873 parsl.dataflow.dflow DEBUG: Task 873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/5af00947-8a83-451c-b94a-be9ec3aa1e56_isrForCrosstalkSources_1125925_24.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/5af00947-8a83-451c-b94a-be9ec3aa1e56_isrForCrosstalkSources_1125925_24.stderr parsl.dataflow.dflow INFO: Task 856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29ea9e784c44dd8f86212c29ac22c84b with result from task 856 parsl.dataflow.dflow DEBUG: Task 856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/fe6b85ac-07b3-4261-a28d-9b0e201633e5_isrForCrosstalkSources_1125903_61.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/fe6b85ac-07b3-4261-a28d-9b0e201633e5_isrForCrosstalkSources_1125903_61.stderr parsl.dataflow.dflow INFO: Task 874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f34ed689c17041e904f1ccbd360f61cf with result from task 874 parsl.dataflow.dflow DEBUG: Task 874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/3b531d86-9b5a-42da-ba58-6bf5255ef070_isrForCrosstalkSources_1125931_38.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/3b531d86-9b5a-42da-ba58-6bf5255ef070_isrForCrosstalkSources_1125931_38.stderr parsl.dataflow.dflow INFO: Task 878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8517eb000135ab4b5fa041cde0204590 with result from task 878 parsl.dataflow.dflow DEBUG: Task 878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/7eb23229-a3e6-43ea-bdb9-6b8ee71027e5_isrForCrosstalkSources_1125923_45.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/7eb23229-a3e6-43ea-bdb9-6b8ee71027e5_isrForCrosstalkSources_1125923_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:36 proc_lsst.multi:146 [INFO] found job 16896709 in provider checkpoint proc_lsst.multi INFO: found job 16896709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b609cce042432b15da5d1d938c073818 with result from task 876 parsl.dataflow.dflow DEBUG: Task 876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/2eb7eee3-a344-4c10-8a29-e23f31b106c7_isrForCrosstalkSources_1125888_11.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/2eb7eee3-a344-4c10-8a29-e23f31b106c7_isrForCrosstalkSources_1125888_11.stderr parsl.dataflow.dflow INFO: Task 855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5b8278284076efe28fbdf2883b09d9e with result from task 855 parsl.dataflow.dflow DEBUG: Task 855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/777b2b9b-2032-4dcd-b0c3-8734ac2738a4_isrForCrosstalkSources_1125946_37.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/777b2b9b-2032-4dcd-b0c3-8734ac2738a4_isrForCrosstalkSources_1125946_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:36 proc_lsst.multi:146 [INFO] found job 16896710 in provider checkpoint proc_lsst.multi INFO: found job 16896710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4477707780b3a08afab06565ca88be22 with result from task 857 parsl.dataflow.dflow DEBUG: Task 857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/0a3bd108-7f50-4327-acba-84c457e58df4_isrForCrosstalkSources_1125892_36.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/0a3bd108-7f50-4327-acba-84c457e58df4_isrForCrosstalkSources_1125892_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:37 proc_lsst.multi:146 [INFO] found job 16896711 in provider checkpoint proc_lsst.multi INFO: found job 16896711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:37 proc_lsst.multi:146 [INFO] found job 16896712 in provider checkpoint proc_lsst.multi INFO: found job 16896712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:38 proc_lsst.multi:146 [INFO] found job 16896713 in provider checkpoint proc_lsst.multi INFO: found job 16896713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:38 proc_lsst.multi:146 [INFO] found job 16896714 in provider checkpoint proc_lsst.multi INFO: found job 16896714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896447 PD 16896448 PD 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 312, active_tasks = 2846 parsl.dataflow.strategy DEBUG: Executor multi has 2846 active tasks, 35/277 running/pending blocks, and 51 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2a: active_blocks 312 >= max_blocks 312 so not scaling out parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22377792571472 2024-03-07 15:11:39 proc_lsst.multi:146 [INFO] found job 53087 in provider local proc_lsst.multi INFO: found job 53087 in provider local 2024-03-07 15:11:39 proc_lsst.multi:146 [INFO] found job 16896385 in provider astro proc_lsst.multi INFO: found job 16896385 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:40 proc_lsst.multi:146 [INFO] found job 16896386 in provider astro proc_lsst.multi INFO: found job 16896386 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:40 proc_lsst.multi:146 [INFO] found job 16896387 in provider astro proc_lsst.multi INFO: found job 16896387 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4f52f20b5078e6b9c1743e3e6d09e98 with result from task 882 parsl.dataflow.dflow DEBUG: Task 882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/7708a0d7-3f40-4828-99d5-6923b71dc600_isrForCrosstalkSources_1125928_36.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/7708a0d7-3f40-4828-99d5-6923b71dc600_isrForCrosstalkSources_1125928_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:41 proc_lsst.multi:146 [INFO] found job 16896388 in provider astro proc_lsst.multi INFO: found job 16896388 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:41 proc_lsst.multi:146 [INFO] found job 16896389 in provider astro proc_lsst.multi INFO: found job 16896389 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:41 proc_lsst.multi:146 [INFO] found job 16896390 in provider astro proc_lsst.multi INFO: found job 16896390 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95f67cc84e42361702fd058b4e5d831c with result from task 883 parsl.dataflow.dflow DEBUG: Task 883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/169d2e0c-eaae-4a73-a51f-27a324df53c4_isrForCrosstalkSources_1125895_43.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/169d2e0c-eaae-4a73-a51f-27a324df53c4_isrForCrosstalkSources_1125895_43.stderr parsl.dataflow.dflow INFO: Task 887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66b716b0a8d2787b20c867fda005bce5 with result from task 887 parsl.dataflow.dflow DEBUG: Task 887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/0c39049a-e2dc-4944-8137-a60f3e465d48_isrForCrosstalkSources_1125895_28.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/0c39049a-e2dc-4944-8137-a60f3e465d48_isrForCrosstalkSources_1125895_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:41 proc_lsst.multi:146 [INFO] found job 16896392 in provider astro proc_lsst.multi INFO: found job 16896392 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:41 proc_lsst.multi:146 [INFO] found job 16896394 in provider astro proc_lsst.multi INFO: found job 16896394 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ba24469e4a208f73734061c4ce88515 with result from task 896 parsl.dataflow.dflow DEBUG: Task 896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/674fee06-a48d-4cec-92c7-89ffba94c004_isrForCrosstalkSources_1125906_44.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/674fee06-a48d-4cec-92c7-89ffba94c004_isrForCrosstalkSources_1125906_44.stderr parsl.dataflow.dflow INFO: Task 891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d07c16a9a4d065687c6547c085fc0c1 with result from task 891 parsl.dataflow.dflow DEBUG: Task 891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c5919648-a0b9-443d-b0bf-4c83ac37255a_isrForCrosstalkSources_1125910_13.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c5919648-a0b9-443d-b0bf-4c83ac37255a_isrForCrosstalkSources_1125910_13.stderr parsl.dataflow.dflow INFO: Task 889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ef56728e5ab21e4ed10a540611e2f6f with result from task 889 parsl.dataflow.dflow DEBUG: Task 889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a5460acc-7e93-4f27-b804-ada308068e0e_isrForCrosstalkSources_1125937_49.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a5460acc-7e93-4f27-b804-ada308068e0e_isrForCrosstalkSources_1125937_49.stderr parsl.dataflow.dflow INFO: Task 848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4526abdc5b4535da94ad0c17577db187 with result from task 848 parsl.dataflow.dflow DEBUG: Task 848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/71fba330-8706-4987-b6d1-5aa97a755acc_isrForCrosstalkSources_1125904_46.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/71fba330-8706-4987-b6d1-5aa97a755acc_isrForCrosstalkSources_1125904_46.stderr parsl.dataflow.dflow INFO: Task 879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c9d9d099e7fef71012e0e190facc928 with result from task 879 parsl.dataflow.dflow DEBUG: Task 879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/b8e907cd-e9ec-442f-9191-0c8d346625b3_isrForCrosstalkSources_1125936_25.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/b8e907cd-e9ec-442f-9191-0c8d346625b3_isrForCrosstalkSources_1125936_25.stderr parsl.dataflow.dflow INFO: Task 894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a5c72ff3a11ec9528e56249192ea5ac with result from task 894 parsl.dataflow.dflow DEBUG: Task 894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/b4643dd2-361b-4b79-8b2c-4d7d61a30b88_isrForCrosstalkSources_1125941_41.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/b4643dd2-361b-4b79-8b2c-4d7d61a30b88_isrForCrosstalkSources_1125941_41.stderr parsl.dataflow.dflow INFO: Task 871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 221333595c22244a00dba5d5688b9fcd with result from task 871 parsl.dataflow.dflow DEBUG: Task 871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/54be4924-32e7-47ab-b414-1058786d89db_isrForCrosstalkSources_1125906_31.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/54be4924-32e7-47ab-b414-1058786d89db_isrForCrosstalkSources_1125906_31.stderr parsl.dataflow.dflow INFO: Task 849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d7f6ff25ba06aaea8d393bf387e84c3 with result from task 849 parsl.dataflow.dflow DEBUG: Task 849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/9aa090db-5ff7-4dd2-a5a9-13c7238dd86c_isrForCrosstalkSources_1125898_6.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/9aa090db-5ff7-4dd2-a5a9-13c7238dd86c_isrForCrosstalkSources_1125898_6.stderr parsl.dataflow.dflow INFO: Task 898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d3198de5ddff8f40156ca94dad33eb3 with result from task 898 parsl.dataflow.dflow DEBUG: Task 898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/2d40dc3d-a1c6-4176-bd47-a459dc861bc8_isrForCrosstalkSources_1125924_10.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/2d40dc3d-a1c6-4176-bd47-a459dc861bc8_isrForCrosstalkSources_1125924_10.stderr parsl.dataflow.dflow INFO: Task 850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5ef7a265f38dd563dba378198a822be with result from task 850 parsl.dataflow.dflow DEBUG: Task 850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/cefad942-580e-4548-86cd-9a61a43b5cb1_isrForCrosstalkSources_1125940_59.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/cefad942-580e-4548-86cd-9a61a43b5cb1_isrForCrosstalkSources_1125940_59.stderr parsl.dataflow.dflow INFO: Task 846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01ca89675a523928d915877cd8f34b94 with result from task 846 parsl.dataflow.dflow DEBUG: Task 846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/50bd657c-0c96-4842-a57c-d7771bccc4fb_isrForCrosstalkSources_1125920_11.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/50bd657c-0c96-4842-a57c-d7771bccc4fb_isrForCrosstalkSources_1125920_11.stderr parsl.dataflow.dflow INFO: Task 847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 178c315b5001e983b57e1b132793ef67 with result from task 847 parsl.dataflow.dflow DEBUG: Task 847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/86f5954e-e5b9-4e32-bae7-72fdbf99f56e_isrForCrosstalkSources_1125896_53.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/86f5954e-e5b9-4e32-bae7-72fdbf99f56e_isrForCrosstalkSources_1125896_53.stderr parsl.dataflow.dflow INFO: Task 886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a1cef4b3735de13874660bc21cf1608 with result from task 886 parsl.dataflow.dflow DEBUG: Task 886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/b7e0b543-051c-435d-b02f-53f55cbccc53_isrForCrosstalkSources_1125897_27.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/b7e0b543-051c-435d-b02f-53f55cbccc53_isrForCrosstalkSources_1125897_27.stderr parsl.dataflow.dflow INFO: Task 854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57a1cc4ce3b2dee511d3c42ae9c6ef56 with result from task 854 parsl.dataflow.dflow DEBUG: Task 854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/245fd55b-de6d-4382-87c3-ecee7280ab12_isrForCrosstalkSources_1125892_55.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/245fd55b-de6d-4382-87c3-ecee7280ab12_isrForCrosstalkSources_1125892_55.stderr parsl.dataflow.dflow INFO: Task 915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84464538c972c613d901981517946cc7 with result from task 915 parsl.dataflow.dflow DEBUG: Task 915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/1676fbd7-8bd1-48c3-bdfd-0e306474e18e_isrForCrosstalkSources_1125930_25.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/1676fbd7-8bd1-48c3-bdfd-0e306474e18e_isrForCrosstalkSources_1125930_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:44 proc_lsst.multi:146 [INFO] found job 16896395 in provider astro proc_lsst.multi INFO: found job 16896395 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14755e04ad8ebdb1c60191567bdbd974 with result from task 914 parsl.dataflow.dflow DEBUG: Task 914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/3a3cf5c5-0cc4-4a2a-8572-a5fc807e164b_isrForCrosstalkSources_1125908_39.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/3a3cf5c5-0cc4-4a2a-8572-a5fc807e164b_isrForCrosstalkSources_1125908_39.stderr parsl.dataflow.dflow INFO: Task 897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2adf4ceac43ce1923673ca7a043a903 with result from task 897 parsl.dataflow.dflow DEBUG: Task 897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/9423b06e-16f6-4fba-8953-1ec04670e990_isrForCrosstalkSources_1125940_45.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/9423b06e-16f6-4fba-8953-1ec04670e990_isrForCrosstalkSources_1125940_45.stderr parsl.dataflow.dflow INFO: Task 905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33875357e31092234e00f660e62877cf with result from task 905 parsl.dataflow.dflow DEBUG: Task 905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/d009bf13-66b3-4bbc-bc74-43621da64982_isrForCrosstalkSources_1125933_31.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/d009bf13-66b3-4bbc-bc74-43621da64982_isrForCrosstalkSources_1125933_31.stderr parsl.dataflow.dflow INFO: Task 908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82b8bc1099a854a97ff0a9ad31604f6f with result from task 908 parsl.dataflow.dflow DEBUG: Task 908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/000ecf68-f3d7-4270-b044-9d82b114f41a_isrForCrosstalkSources_1125928_47.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/000ecf68-f3d7-4270-b044-9d82b114f41a_isrForCrosstalkSources_1125928_47.stderr parsl.dataflow.dflow INFO: Task 893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45717dcaf9942b77e60fc1ec8eb9ab56 with result from task 893 parsl.dataflow.dflow DEBUG: Task 893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/77182f57-0547-4042-8ce7-a3430f23e54d_isrForCrosstalkSources_1125897_52.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/77182f57-0547-4042-8ce7-a3430f23e54d_isrForCrosstalkSources_1125897_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:44 proc_lsst.multi:146 [INFO] found job 16896396 in provider astro proc_lsst.multi INFO: found job 16896396 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7013537015c6565046b8f617d13ae59 with result from task 890 parsl.dataflow.dflow DEBUG: Task 890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/22cf15bc-7783-49ce-9204-9908ab6b18fa_isrForCrosstalkSources_1125946_3.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/22cf15bc-7783-49ce-9204-9908ab6b18fa_isrForCrosstalkSources_1125946_3.stderr parsl.dataflow.dflow INFO: Task 885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e3c73dd308f5462b97f9721b310eb9d with result from task 885 parsl.dataflow.dflow DEBUG: Task 885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/b134820b-0147-46ed-96ea-21580e36afd3_isrForCrosstalkSources_1125896_12.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/b134820b-0147-46ed-96ea-21580e36afd3_isrForCrosstalkSources_1125896_12.stderr parsl.dataflow.dflow INFO: Task 884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f5e47085069f39e48fd749498137984 with result from task 884 parsl.dataflow.dflow DEBUG: Task 884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/c270873f-d7d3-48fa-a2d0-003e3151df64_isrForCrosstalkSources_1125897_62.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/c270873f-d7d3-48fa-a2d0-003e3151df64_isrForCrosstalkSources_1125897_62.stderr parsl.dataflow.dflow INFO: Task 888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eff42acce0e163fd44f9f82dab2e7307 with result from task 888 parsl.dataflow.dflow DEBUG: Task 888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/823864f8-0bbb-4531-99cf-8dde03b2fdc4_isrForCrosstalkSources_1125907_31.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/823864f8-0bbb-4531-99cf-8dde03b2fdc4_isrForCrosstalkSources_1125907_31.stderr parsl.dataflow.dflow INFO: Task 892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3410dd5f6e103a59a8e7390efdfc61a with result from task 892 parsl.dataflow.dflow DEBUG: Task 892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/672f59a1-9a80-47cc-90aa-01b0bba548ff_isrForCrosstalkSources_1125929_9.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/672f59a1-9a80-47cc-90aa-01b0bba548ff_isrForCrosstalkSources_1125929_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:44 proc_lsst.multi:146 [INFO] found job 16896397 in provider astro proc_lsst.multi INFO: found job 16896397 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:45 proc_lsst.multi:146 [INFO] found job 16896398 in provider astro proc_lsst.multi INFO: found job 16896398 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b97f7da7f2c2da10494e4650a02c5c24 with result from task 901 parsl.dataflow.dflow DEBUG: Task 901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/1e6c2f67-5a13-4dde-b157-4ab5068b5896_isrForCrosstalkSources_1125894_23.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/1e6c2f67-5a13-4dde-b157-4ab5068b5896_isrForCrosstalkSources_1125894_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:45 proc_lsst.multi:146 [INFO] found job 16896399 in provider astro proc_lsst.multi INFO: found job 16896399 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8945b0854b9810ab408dba659fa78e6a with result from task 922 parsl.dataflow.dflow DEBUG: Task 922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/8bd58d55-9460-43fa-a66c-a025f66be7c6_isrForCrosstalkSources_1125897_41.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/8bd58d55-9460-43fa-a66c-a025f66be7c6_isrForCrosstalkSources_1125897_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:46 proc_lsst.multi:146 [INFO] found job 16896400 in provider astro proc_lsst.multi INFO: found job 16896400 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7235c5e2667aed1a759dbb516aad8179 with result from task 921 parsl.dataflow.dflow DEBUG: Task 921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/2a86149f-bee1-49d0-a718-8c86585cd92e_isrForCrosstalkSources_1125892_50.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/2a86149f-bee1-49d0-a718-8c86585cd92e_isrForCrosstalkSources_1125892_50.stderr parsl.dataflow.dflow INFO: Task 902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 246d43f6ff5440acb39f247ac4fe5fbb with result from task 902 parsl.dataflow.dflow DEBUG: Task 902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3d32d322-4195-4a46-9b06-417bbe7f754e_isrForCrosstalkSources_1125941_14.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3d32d322-4195-4a46-9b06-417bbe7f754e_isrForCrosstalkSources_1125941_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:46 proc_lsst.multi:146 [INFO] found job 16896401 in provider astro proc_lsst.multi INFO: found job 16896401 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10e215fc53dc71b297eef943256f985e with result from task 913 parsl.dataflow.dflow DEBUG: Task 913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/0bcb33e5-01b4-4b2b-97eb-395e0b494e4c_isrForCrosstalkSources_1125918_21.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/0bcb33e5-01b4-4b2b-97eb-395e0b494e4c_isrForCrosstalkSources_1125918_21.stderr parsl.dataflow.dflow INFO: Task 919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a557947786688e0972b5d44f60ed8167 with result from task 919 parsl.dataflow.dflow DEBUG: Task 919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/58df12de-83a8-447a-a277-5ddd12fda3ce_isrForCrosstalkSources_1125904_51.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/58df12de-83a8-447a-a277-5ddd12fda3ce_isrForCrosstalkSources_1125904_51.stderr parsl.dataflow.dflow INFO: Task 899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e9807696f70ea2880c5be141369dba3 with result from task 899 parsl.dataflow.dflow DEBUG: Task 899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/80ccae37-d16c-4ded-8581-a45fb7209806_isrForCrosstalkSources_1125938_16.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/80ccae37-d16c-4ded-8581-a45fb7209806_isrForCrosstalkSources_1125938_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:46 proc_lsst.multi:146 [INFO] found job 16896402 in provider astro proc_lsst.multi INFO: found job 16896402 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64cb5cd0ebbbf0c05bd85600ef9af7ce with result from task 903 parsl.dataflow.dflow DEBUG: Task 903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/e89f3755-8671-43d8-9c35-cd2087d42897_isrForCrosstalkSources_1125899_8.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/e89f3755-8671-43d8-9c35-cd2087d42897_isrForCrosstalkSources_1125899_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:47 proc_lsst.multi:146 [INFO] found job 16896403 in provider astro proc_lsst.multi INFO: found job 16896403 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcdf338cf4b0edd3783b2572ff62d079 with result from task 907 parsl.dataflow.dflow DEBUG: Task 907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/47a08e0e-78dc-45ed-a56c-3067a6a6f81e_isrForCrosstalkSources_1125892_24.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/47a08e0e-78dc-45ed-a56c-3067a6a6f81e_isrForCrosstalkSources_1125892_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:47 proc_lsst.multi:146 [INFO] found job 16896404 in provider astro proc_lsst.multi INFO: found job 16896404 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c17a13d30294b10fff1c983965cd19d with result from task 904 parsl.dataflow.dflow DEBUG: Task 904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/7d10d1d5-fc3c-4bce-a411-879f5180c906_isrForCrosstalkSources_1125897_20.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/7d10d1d5-fc3c-4bce-a411-879f5180c906_isrForCrosstalkSources_1125897_20.stderr parsl.dataflow.dflow INFO: Task 924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdf228bc9c913e29cfeeab3a9bdb96cf with result from task 924 parsl.dataflow.dflow DEBUG: Task 924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/8f795c4b-ee5d-44da-8301-74cf599e60c9_isrForCrosstalkSources_1125910_16.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/8f795c4b-ee5d-44da-8301-74cf599e60c9_isrForCrosstalkSources_1125910_16.stderr parsl.dataflow.dflow INFO: Task 918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04a6f8a5f70882acfdbbd6fb6d519b3e with result from task 918 parsl.dataflow.dflow DEBUG: Task 918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/93df4af7-b3cb-4860-bf1b-ba064182c48b_isrForCrosstalkSources_1125935_51.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/93df4af7-b3cb-4860-bf1b-ba064182c48b_isrForCrosstalkSources_1125935_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:47 proc_lsst.multi:146 [INFO] found job 16896405 in provider astro proc_lsst.multi INFO: found job 16896405 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46334d3dc29fed0fb4f0873006c590bb with result from task 909 parsl.dataflow.dflow DEBUG: Task 909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/b19597a6-01d7-4adc-b929-36a48f2a5090_isrForCrosstalkSources_1125889_18.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/b19597a6-01d7-4adc-b929-36a48f2a5090_isrForCrosstalkSources_1125889_18.stderr parsl.dataflow.dflow INFO: Task 912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 304aee88738e3db1fc3623356fa520b9 with result from task 912 parsl.dataflow.dflow DEBUG: Task 912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/53ba9cbd-ca01-48ec-b531-3a32d73f202c_isrForCrosstalkSources_1125898_29.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/53ba9cbd-ca01-48ec-b531-3a32d73f202c_isrForCrosstalkSources_1125898_29.stderr parsl.dataflow.dflow INFO: Task 906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10d23a04ac2e508ea4b4033e181164cb with result from task 906 parsl.dataflow.dflow DEBUG: Task 906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/c5967a72-9980-4054-9e5a-1710482124c6_isrForCrosstalkSources_1125901_16.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/c5967a72-9980-4054-9e5a-1710482124c6_isrForCrosstalkSources_1125901_16.stderr parsl.dataflow.dflow INFO: Task 917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0392971dcd08859e741e28d9ee7837a with result from task 917 parsl.dataflow.dflow DEBUG: Task 917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/4ca0315d-1d2d-446e-9a3c-8b442dca842d_isrForCrosstalkSources_1125889_49.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/4ca0315d-1d2d-446e-9a3c-8b442dca842d_isrForCrosstalkSources_1125889_49.stderr parsl.dataflow.dflow INFO: Task 923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0aab929b45e9f6ce0b661e7c9ea84d1 with result from task 923 parsl.dataflow.dflow DEBUG: Task 923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/6c3d9194-e7ec-4f0b-9d93-b456fc846f49_isrForCrosstalkSources_1125933_41.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/6c3d9194-e7ec-4f0b-9d93-b456fc846f49_isrForCrosstalkSources_1125933_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:48 proc_lsst.multi:146 [INFO] found job 16896408 in provider astro proc_lsst.multi INFO: found job 16896408 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc86bffc8e56094481904d5024030525 with result from task 895 parsl.dataflow.dflow DEBUG: Task 895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a78e75a0-1013-430f-9bfe-e4d44eaa8edb_isrForCrosstalkSources_1125921_50.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a78e75a0-1013-430f-9bfe-e4d44eaa8edb_isrForCrosstalkSources_1125921_50.stderr parsl.dataflow.dflow INFO: Task 900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c156a72ab2c403e15952b00d32c97800 with result from task 900 parsl.dataflow.dflow DEBUG: Task 900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/c98c0fd4-21e8-4862-bd26-102830cd844a_isrForCrosstalkSources_1125937_58.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/c98c0fd4-21e8-4862-bd26-102830cd844a_isrForCrosstalkSources_1125937_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:48 proc_lsst.multi:146 [INFO] found job 16896409 in provider astro proc_lsst.multi INFO: found job 16896409 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aaf29e800979e7f3fc721ca203543561 with result from task 916 parsl.dataflow.dflow DEBUG: Task 916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/b460cd4a-0aec-4964-9133-6e5a9753fc61_isrForCrosstalkSources_1125940_58.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/b460cd4a-0aec-4964-9133-6e5a9753fc61_isrForCrosstalkSources_1125940_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:48 proc_lsst.multi:146 [INFO] found job 16896410 in provider astro proc_lsst.multi INFO: found job 16896410 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fd7ed24b50a4b054fea64d6a594a7f9 with result from task 911 parsl.dataflow.dflow DEBUG: Task 911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/ec298cf7-6ac1-4f2e-8a03-4175406ccdf7_isrForCrosstalkSources_1125904_47.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/ec298cf7-6ac1-4f2e-8a03-4175406ccdf7_isrForCrosstalkSources_1125904_47.stderr parsl.dataflow.dflow INFO: Task 910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f30b88fe3f23413dba197a33a5aafc4f with result from task 910 parsl.dataflow.dflow DEBUG: Task 910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d8c26114-bfbd-4431-97b3-0e7725af85e2_isrForCrosstalkSources_1125923_61.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d8c26114-bfbd-4431-97b3-0e7725af85e2_isrForCrosstalkSources_1125923_61.stderr parsl.dataflow.dflow INFO: Task 920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85e3bf7636c6a604aa3524d6bb7c483e with result from task 920 parsl.dataflow.dflow DEBUG: Task 920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/57092952-f124-41c8-a69c-d4dd3498d960_isrForCrosstalkSources_1125916_62.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/57092952-f124-41c8-a69c-d4dd3498d960_isrForCrosstalkSources_1125916_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:49 proc_lsst.multi:146 [INFO] found job 16896411 in provider astro proc_lsst.multi INFO: found job 16896411 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:49 proc_lsst.multi:146 [INFO] found job 16896412 in provider astro proc_lsst.multi INFO: found job 16896412 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55403c42407510ef57961d31921a690e with result from task 928 parsl.dataflow.dflow DEBUG: Task 928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f5ea4ef4-acd7-4adf-9cf4-f918c8cc1094_isrForCrosstalkSources_1125894_35.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f5ea4ef4-acd7-4adf-9cf4-f918c8cc1094_isrForCrosstalkSources_1125894_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:50 proc_lsst.multi:146 [INFO] found job 16896413 in provider astro proc_lsst.multi INFO: found job 16896413 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:50 proc_lsst.multi:146 [INFO] found job 16896414 in provider astro proc_lsst.multi INFO: found job 16896414 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:51 proc_lsst.multi:146 [INFO] found job 16896415 in provider astro proc_lsst.multi INFO: found job 16896415 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:51 proc_lsst.multi:146 [INFO] found job 16896416 in provider astro proc_lsst.multi INFO: found job 16896416 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94f6f7e77e57ce63959d7c78768381cd with result from task 927 parsl.dataflow.dflow DEBUG: Task 927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/5511e4de-58f5-46c4-87c8-e90e75bd3194_isrForCrosstalkSources_1125947_32.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/5511e4de-58f5-46c4-87c8-e90e75bd3194_isrForCrosstalkSources_1125947_32.stderr parsl.dataflow.dflow INFO: Task 930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 582dccb307be369b83ffdd8b80f69654 with result from task 930 parsl.dataflow.dflow DEBUG: Task 930 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/8c858ddb-3f30-4d39-a53b-36862e8d3de0_isrForCrosstalkSources_1125895_39.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/8c858ddb-3f30-4d39-a53b-36862e8d3de0_isrForCrosstalkSources_1125895_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:51 proc_lsst.multi:146 [INFO] found job 16896417 in provider astro proc_lsst.multi INFO: found job 16896417 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdbf7f127771d8a98f3f4f9f0b2809f2 with result from task 926 parsl.dataflow.dflow DEBUG: Task 926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/9ad9a741-ce44-491e-92ec-ea0669533eae_isrForCrosstalkSources_1125931_18.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/9ad9a741-ce44-491e-92ec-ea0669533eae_isrForCrosstalkSources_1125931_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:51 proc_lsst.multi:146 [INFO] found job 16896418 in provider astro proc_lsst.multi INFO: found job 16896418 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52f0183122a0593f0959e0d645b5e424 with result from task 929 parsl.dataflow.dflow DEBUG: Task 929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c70c1e5c-68fe-408d-88f5-307425106330_isrForCrosstalkSources_1125917_28.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c70c1e5c-68fe-408d-88f5-307425106330_isrForCrosstalkSources_1125917_28.stderr parsl.dataflow.dflow INFO: Task 933 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cca5fd9e90c9191553585fdda5a8b46c with result from task 933 parsl.dataflow.dflow DEBUG: Task 933 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/4186c740-def9-447f-aa03-fc5bcebc03d2_isrForCrosstalkSources_1125911_11.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/4186c740-def9-447f-aa03-fc5bcebc03d2_isrForCrosstalkSources_1125911_11.stderr parsl.dataflow.dflow INFO: Task 942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5df3e85f1b06318c509d9abf2d449b8b with result from task 942 parsl.dataflow.dflow DEBUG: Task 942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/81124ccb-14e4-4cf0-9750-7b0d241ca057_isrForCrosstalkSources_1125943_31.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/81124ccb-14e4-4cf0-9750-7b0d241ca057_isrForCrosstalkSources_1125943_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:52 proc_lsst.multi:146 [INFO] found job 16896419 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896419 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.dataflow.dflow INFO: Task 943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d45d117ab4e761baa4210626a662c28 with result from task 943 parsl.dataflow.dflow DEBUG: Task 943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/0db647db-9e8e-4f2d-b2d0-01bb5a3bae8f_isrForCrosstalkSources_1125925_12.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/0db647db-9e8e-4f2d-b2d0-01bb5a3bae8f_isrForCrosstalkSources_1125925_12.stderr parsl.dataflow.dflow INFO: Task 936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af7f94ecbc711ac62831caa783a099f5 with result from task 936 parsl.dataflow.dflow DEBUG: Task 936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/eb5ffebc-2d5f-4312-a59a-c34b50770f15_isrForCrosstalkSources_1125907_62.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/eb5ffebc-2d5f-4312-a59a-c34b50770f15_isrForCrosstalkSources_1125907_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:53 proc_lsst.multi:146 [INFO] found job 16896420 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896420 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.dataflow.dflow INFO: Task 946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 088579552df83940b2abced3300b98a8 with result from task 946 parsl.dataflow.dflow DEBUG: Task 946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0d87e29d-78dd-40cf-bf8e-561f4c7bc183_isrForCrosstalkSources_1125890_13.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0d87e29d-78dd-40cf-bf8e-561f4c7bc183_isrForCrosstalkSources_1125890_13.stderr parsl.dataflow.dflow INFO: Task 925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b3b37b7ad22e0c61da8c388890f2994 with result from task 925 parsl.dataflow.dflow DEBUG: Task 925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/538b0d68-b701-4813-af9f-ec78fe368713_isrForCrosstalkSources_1125903_42.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/538b0d68-b701-4813-af9f-ec78fe368713_isrForCrosstalkSources_1125903_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:53 proc_lsst.multi:146 [INFO] found job 16896421 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896421 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.dataflow.dflow INFO: Task 945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d2c0999bfe5cfc117c72c78a49f21d8 with result from task 945 parsl.dataflow.dflow DEBUG: Task 945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/c166e308-38c0-42fe-94c1-a9d7ac6cafdd_isrForCrosstalkSources_1125909_38.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/c166e308-38c0-42fe-94c1-a9d7ac6cafdd_isrForCrosstalkSources_1125909_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:53 proc_lsst.multi:146 [INFO] found job 16896422 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896422 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:54 proc_lsst.multi:146 [INFO] found job 16896423 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896423 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:54 proc_lsst.multi:146 [INFO] found job 16896424 in provider escience_a40 proc_lsst.multi INFO: found job 16896424 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10fb0853804e1bbbf003006d4cc4c36f with result from task 944 parsl.dataflow.dflow DEBUG: Task 944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/59994020-8f18-42ad-8ada-507b2c22327f_isrForCrosstalkSources_1125946_8.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/59994020-8f18-42ad-8ada-507b2c22327f_isrForCrosstalkSources_1125946_8.stderr parsl.dataflow.dflow INFO: Task 940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 238aff4f2ea11514720804b87dbe738e with result from task 940 parsl.dataflow.dflow DEBUG: Task 940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/99ad0f22-d1d9-4acf-9237-820473489150_isrForCrosstalkSources_1125908_53.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/99ad0f22-d1d9-4acf-9237-820473489150_isrForCrosstalkSources_1125908_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:55 proc_lsst.multi:146 [INFO] found job 16896425 in provider escience_a40 proc_lsst.multi INFO: found job 16896425 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:55 proc_lsst.multi:146 [INFO] found job 16896426 in provider escience_a40 proc_lsst.multi INFO: found job 16896426 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 950 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6884a5ec0240ec13a47630383b326834 with result from task 950 parsl.dataflow.dflow DEBUG: Task 950 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/3454be08-e5de-4c3c-ac23-24fba7b98625_isrForCrosstalkSources_1125945_25.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/3454be08-e5de-4c3c-ac23-24fba7b98625_isrForCrosstalkSources_1125945_25.stderr parsl.dataflow.dflow INFO: Task 951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27f26597af115a37dc4b966c15a24e7a with result from task 951 parsl.dataflow.dflow DEBUG: Task 951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/f9571479-3379-4fc5-aef9-98a0c9d6e9ac_isrForCrosstalkSources_1125890_32.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/f9571479-3379-4fc5-aef9-98a0c9d6e9ac_isrForCrosstalkSources_1125890_32.stderr parsl.dataflow.dflow INFO: Task 955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59d602aadd18ecc956499bacebffedb7 with result from task 955 parsl.dataflow.dflow DEBUG: Task 955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/3c5c1a06-0892-422c-b0ca-e5307ff3f6a8_isrForCrosstalkSources_1125910_4.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/3c5c1a06-0892-422c-b0ca-e5307ff3f6a8_isrForCrosstalkSources_1125910_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:55 proc_lsst.multi:146 [INFO] found job 16896427 in provider escience_a40 proc_lsst.multi INFO: found job 16896427 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7db4a07bf2f1612c1f6c954b95c7fa6f with result from task 931 parsl.dataflow.dflow DEBUG: Task 931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/54421c41-f0f3-46ca-9e91-1d603e937167_isrForCrosstalkSources_1125894_32.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/54421c41-f0f3-46ca-9e91-1d603e937167_isrForCrosstalkSources_1125894_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:56 proc_lsst.multi:146 [INFO] found job 16896428 in provider escience_a40 proc_lsst.multi INFO: found job 16896428 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38197dd447d66dfbeef2f3de097b0583 with result from task 954 parsl.dataflow.dflow DEBUG: Task 954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/4743aa7f-31e0-46f4-8169-4df6b500f067_isrForCrosstalkSources_1125945_14.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/4743aa7f-31e0-46f4-8169-4df6b500f067_isrForCrosstalkSources_1125945_14.stderr parsl.dataflow.dflow INFO: Task 952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d1ca88ad5295bd22df44c48f3ed2e23 with result from task 952 parsl.dataflow.dflow DEBUG: Task 952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/a5244f68-8152-44a7-8685-36d7013c22b3_isrForCrosstalkSources_1125910_23.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/a5244f68-8152-44a7-8685-36d7013c22b3_isrForCrosstalkSources_1125910_23.stderr parsl.dataflow.dflow INFO: Task 941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19fd82c19e5436b853e09448d75939ef with result from task 941 parsl.dataflow.dflow DEBUG: Task 941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/6d11d244-79bc-4cec-95b2-d6c097804076_isrForCrosstalkSources_1125944_21.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/6d11d244-79bc-4cec-95b2-d6c097804076_isrForCrosstalkSources_1125944_21.stderr parsl.dataflow.dflow INFO: Task 948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 666939dd2cd22a38d65a19fb08c19530 with result from task 948 parsl.dataflow.dflow DEBUG: Task 948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/620774ff-7652-4d51-bc88-5a5dfb15bd6d_isrForCrosstalkSources_1125946_57.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/620774ff-7652-4d51-bc88-5a5dfb15bd6d_isrForCrosstalkSources_1125946_57.stderr parsl.dataflow.dflow INFO: Task 938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87d9631680ca7337251e8270755ad5d0 with result from task 938 parsl.dataflow.dflow DEBUG: Task 938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/396e02df-112c-4a17-a8fb-fa5da178771e_isrForCrosstalkSources_1125941_13.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/396e02df-112c-4a17-a8fb-fa5da178771e_isrForCrosstalkSources_1125941_13.stderr parsl.dataflow.dflow INFO: Task 947 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c498fbcdfcfdfdc0f173422e2a409835 with result from task 947 parsl.dataflow.dflow DEBUG: Task 947 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8bcfb6b2-e09b-45c3-830c-40ac2ce6d9c9_isrForCrosstalkSources_1125944_50.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8bcfb6b2-e09b-45c3-830c-40ac2ce6d9c9_isrForCrosstalkSources_1125944_50.stderr parsl.dataflow.dflow INFO: Task 934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c0527777000ff8069e7fb7f8d5cddaa with result from task 934 parsl.dataflow.dflow DEBUG: Task 934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/e36b22f3-6d70-43af-960d-bbab4d61ca2c_isrForCrosstalkSources_1125888_45.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/e36b22f3-6d70-43af-960d-bbab4d61ca2c_isrForCrosstalkSources_1125888_45.stderr parsl.dataflow.dflow INFO: Task 953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00ac38038ccf4d5f7f1a553a54957e97 with result from task 953 parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:56 proc_lsst.multi:146 [INFO] found job 16896429 in provider escience_a40 proc_lsst.multi INFO: found job 16896429 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow DEBUG: Task 953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/9709dcb6-2f49-4f1e-9f17-c0155f104f00_isrForCrosstalkSources_1125927_3.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/9709dcb6-2f49-4f1e-9f17-c0155f104f00_isrForCrosstalkSources_1125927_3.stderr parsl.dataflow.dflow INFO: Task 935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c204ac21f03d73a6af17e0b44dec4a25 with result from task 935 parsl.dataflow.dflow DEBUG: Task 935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/567a6aa1-5c85-45ea-8a9c-0ed9addcb87b_isrForCrosstalkSources_1125914_9.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/567a6aa1-5c85-45ea-8a9c-0ed9addcb87b_isrForCrosstalkSources_1125914_9.stderr parsl.dataflow.dflow INFO: Task 939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9fe846df143167826033c3e8d8b4749 with result from task 939 parsl.dataflow.dflow DEBUG: Task 939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/ddd87100-d3c0-447a-965c-fa186a7263e5_isrForCrosstalkSources_1125897_11.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/ddd87100-d3c0-447a-965c-fa186a7263e5_isrForCrosstalkSources_1125897_11.stderr parsl.dataflow.dflow INFO: Task 937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b89dba6e6c26f3b619015530bc310c26 with result from task 937 parsl.dataflow.dflow DEBUG: Task 937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/4527265d-699d-4ebf-a396-1cdc36f811db_isrForCrosstalkSources_1125897_38.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/4527265d-699d-4ebf-a396-1cdc36f811db_isrForCrosstalkSources_1125897_38.stderr parsl.dataflow.dflow INFO: Task 949 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 048fda621c95d602653756e9ab807283 with result from task 949 parsl.dataflow.dflow DEBUG: Task 949 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/7f006787-353c-43e4-b220-dd3a7b9008ce_isrForCrosstalkSources_1125900_41.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/7f006787-353c-43e4-b220-dd3a7b9008ce_isrForCrosstalkSources_1125900_41.stderr parsl.dataflow.dflow INFO: Task 932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c14f641291baa6d9c5c80d9a19328ff8 with result from task 932 parsl.dataflow.dflow DEBUG: Task 932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/5e81597f-217c-4dbd-a589-02179455f9d5_isrForCrosstalkSources_1125902_2.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/5e81597f-217c-4dbd-a589-02179455f9d5_isrForCrosstalkSources_1125902_2.stderr parsl.dataflow.dflow INFO: Task 963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1df88ed6d2212eec0e7b52cd384316e2 with result from task 963 parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:57 proc_lsst.multi:146 [INFO] found job 16896430 in provider escience_a40 proc_lsst.multi INFO: found job 16896430 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow DEBUG: Task 963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/5e9ba6f6-3cb7-4699-ae2c-9e6be647934d_isrForCrosstalkSources_1125944_29.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/5e9ba6f6-3cb7-4699-ae2c-9e6be647934d_isrForCrosstalkSources_1125944_29.stderr parsl.dataflow.dflow INFO: Task 969 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5385c93cb277fa3f27f923c381791adf with result from task 969 parsl.dataflow.dflow DEBUG: Task 969 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/f6493a5a-6826-43f4-af98-187dfb13c85e_isrForCrosstalkSources_1125924_17.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/f6493a5a-6826-43f4-af98-187dfb13c85e_isrForCrosstalkSources_1125924_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:57 proc_lsst.multi:146 [INFO] found job 16896431 in provider escience_a40 proc_lsst.multi INFO: found job 16896431 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:58 proc_lsst.multi:146 [INFO] found job 16896432 in provider escience_a40 proc_lsst.multi INFO: found job 16896432 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:58 proc_lsst.multi:146 [INFO] found job 16896433 in provider escience_a40 proc_lsst.multi INFO: found job 16896433 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a72de7ae505ed7625cc18948185c0775 with result from task 968 parsl.dataflow.dflow DEBUG: Task 968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/d757fab5-3dbd-4614-90bd-0cd6657df766_isrForCrosstalkSources_1125896_4.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/d757fab5-3dbd-4614-90bd-0cd6657df766_isrForCrosstalkSources_1125896_4.stderr parsl.dataflow.dflow INFO: Task 957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 572328a3104cffeddd658de71b6e7224 with result from task 957 parsl.dataflow.dflow DEBUG: Task 957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/66b05aee-8953-4e03-8be9-d3400cbc97ec_isrForCrosstalkSources_1125888_16.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/66b05aee-8953-4e03-8be9-d3400cbc97ec_isrForCrosstalkSources_1125888_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:59 proc_lsst.multi:146 [INFO] found job 16896434 in provider escience_a40 proc_lsst.multi INFO: found job 16896434 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b4beef6ea9a18fc7c0631ff82069851 with result from task 975 parsl.dataflow.dflow DEBUG: Task 975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/16dea266-3a0f-486e-a50c-28cbee3c44d1_isrForCrosstalkSources_1125903_14.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/16dea266-3a0f-486e-a50c-28cbee3c44d1_isrForCrosstalkSources_1125903_14.stderr parsl.dataflow.dflow INFO: Task 976 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 607431e81ef65a0cc2b2c7457e3b0515 with result from task 976 parsl.dataflow.dflow DEBUG: Task 976 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/a98c7f0a-4163-4e63-aa56-77518e9e350c_isrForCrosstalkSources_1125931_61.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/a98c7f0a-4163-4e63-aa56-77518e9e350c_isrForCrosstalkSources_1125931_61.stderr parsl.dataflow.dflow INFO: Task 962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f8333c23dca5c39bc5b1c2d6c7f19ed with result from task 962 parsl.dataflow.dflow DEBUG: Task 962 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/d6e615fe-477c-408a-8ef6-c458e3a903a2_isrForCrosstalkSources_1125922_7.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/d6e615fe-477c-408a-8ef6-c458e3a903a2_isrForCrosstalkSources_1125922_7.stderr parsl.dataflow.dflow INFO: Task 972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22c2e134b33b5c0c863fe80a0c11d7ff with result from task 972 parsl.dataflow.dflow DEBUG: Task 972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/1c2d13d7-2a8d-4382-b72d-10271f756c2b_isrForCrosstalkSources_1125943_20.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/1c2d13d7-2a8d-4382-b72d-10271f756c2b_isrForCrosstalkSources_1125943_20.stderr parsl.dataflow.dflow INFO: Task 961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 468f0e511307f77f3e260aed03af193f with result from task 961 parsl.dataflow.dflow DEBUG: Task 961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/7d0e4d18-262d-4e2f-af01-73e933c8568f_isrForCrosstalkSources_1125894_51.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/7d0e4d18-262d-4e2f-af01-73e933c8568f_isrForCrosstalkSources_1125894_51.stderr parsl.dataflow.dflow INFO: Task 956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 154e6f7ff17c7ee6a35d3d2735f85df1 with result from task 956 parsl.dataflow.dflow DEBUG: Task 956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/492665a7-0c30-4f63-b292-7d07f8536cf2_isrForCrosstalkSources_1125915_16.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/492665a7-0c30-4f63-b292-7d07f8536cf2_isrForCrosstalkSources_1125915_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:11:59 proc_lsst.multi:146 [INFO] found job 16896435 in provider escience_a40 proc_lsst.multi INFO: found job 16896435 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 964 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02a5b9ca49cd7feb0ab5ffa5bd1ad37b with result from task 964 parsl.dataflow.dflow DEBUG: Task 964 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/97ef57b5-d20f-4798-aaca-d2da90220b49_isrForCrosstalkSources_1125921_51.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/97ef57b5-d20f-4798-aaca-d2da90220b49_isrForCrosstalkSources_1125921_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:00 proc_lsst.multi:146 [INFO] found job 16896437 in provider escience_a40 proc_lsst.multi INFO: found job 16896437 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 967 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72c1ba2467ec23ec4c976cb37bcaac17 with result from task 967 parsl.dataflow.dflow DEBUG: Task 967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/f501ff70-c9da-4c8c-b490-10412234c39b_isrForCrosstalkSources_1125900_28.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/f501ff70-c9da-4c8c-b490-10412234c39b_isrForCrosstalkSources_1125900_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:00 proc_lsst.multi:146 [INFO] found job 16896438 in provider escience_a40 proc_lsst.multi INFO: found job 16896438 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dfb30a8e6504c3d329f78eb3bcbdd56 with result from task 965 parsl.dataflow.dflow DEBUG: Task 965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/ef1c2832-0a41-40ba-8720-20e59b664227_isrForCrosstalkSources_1125905_41.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/ef1c2832-0a41-40ba-8720-20e59b664227_isrForCrosstalkSources_1125905_41.stderr parsl.dataflow.dflow INFO: Task 973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8eac6534ed26b4aa605f36c93aada078 with result from task 973 parsl.dataflow.dflow DEBUG: Task 973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/2b64fdc1-f03b-4cc6-acf7-e38ea0722208_isrForCrosstalkSources_1125888_30.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/2b64fdc1-f03b-4cc6-acf7-e38ea0722208_isrForCrosstalkSources_1125888_30.stderr parsl.dataflow.dflow INFO: Task 982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40f4b62382e5ff877a00ccd2909588fd with result from task 982 parsl.dataflow.dflow DEBUG: Task 982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/8dd6ab10-965f-44fc-aa70-97f5d3f6c05b_isrForCrosstalkSources_1125912_1.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/8dd6ab10-965f-44fc-aa70-97f5d3f6c05b_isrForCrosstalkSources_1125912_1.stderr parsl.dataflow.dflow INFO: Task 978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a15e8fa4159ac0589e9c740944c01a91 with result from task 978 parsl.dataflow.dflow DEBUG: Task 978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/94adf543-569d-423c-8541-6464c00bb64c_isrForCrosstalkSources_1125906_7.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/94adf543-569d-423c-8541-6464c00bb64c_isrForCrosstalkSources_1125906_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:00 proc_lsst.multi:146 [INFO] found job 16896439 in provider escience_a40 proc_lsst.multi INFO: found job 16896439 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f6b43999f3fba480fbe09bcd148bcae with result from task 981 parsl.dataflow.dflow DEBUG: Task 981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b9b84359-8784-402c-bc24-a0a3606b46cb_isrForCrosstalkSources_1125934_25.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b9b84359-8784-402c-bc24-a0a3606b46cb_isrForCrosstalkSources_1125934_25.stderr parsl.dataflow.dflow INFO: Task 983 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8801fb29d4643d34b46eb53ebc3ad13 with result from task 983 parsl.dataflow.dflow DEBUG: Task 983 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/27bd975c-f176-4652-81b2-86bbdf7dd5c7_isrForCrosstalkSources_1125933_3.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/27bd975c-f176-4652-81b2-86bbdf7dd5c7_isrForCrosstalkSources_1125933_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:01 proc_lsst.multi:146 [INFO] found job 16896440 in provider escience_a40 proc_lsst.multi INFO: found job 16896440 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e42d9f032dbb141ef3098f54255f59d5 with result from task 959 parsl.dataflow.dflow DEBUG: Task 959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3c46968a-6985-4854-ac91-42d53d67a573_isrForCrosstalkSources_1125912_60.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3c46968a-6985-4854-ac91-42d53d67a573_isrForCrosstalkSources_1125912_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:01 proc_lsst.multi:146 [INFO] found job 16896441 in provider escience_a40 proc_lsst.multi INFO: found job 16896441 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16df4b190dee9a378d3c4d8937101c51 with result from task 960 parsl.dataflow.dflow DEBUG: Task 960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/3dfbf906-85e7-4a87-894f-45cdb6e653de_isrForCrosstalkSources_1125922_50.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/3dfbf906-85e7-4a87-894f-45cdb6e653de_isrForCrosstalkSources_1125922_50.stderr parsl.dataflow.dflow INFO: Task 970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58f335382572526c47112318033265ee with result from task 970 parsl.dataflow.dflow DEBUG: Task 970 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/a7d4c9e3-a645-457c-92a0-0cca47049c34_isrForCrosstalkSources_1125927_47.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/a7d4c9e3-a645-457c-92a0-0cca47049c34_isrForCrosstalkSources_1125927_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:01 proc_lsst.multi:146 [INFO] found job 16896442 in provider escience_a40 proc_lsst.multi INFO: found job 16896442 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 697cfccd430445f26370b25fda07d2e1 with result from task 958 parsl.dataflow.dflow DEBUG: Task 958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/2c791773-fd38-44b8-b071-f047679bc253_isrForCrosstalkSources_1125904_50.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/2c791773-fd38-44b8-b071-f047679bc253_isrForCrosstalkSources_1125904_50.stderr parsl.dataflow.dflow INFO: Task 980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f5adfc577683dbe06eb25324c339358 with result from task 980 parsl.dataflow.dflow DEBUG: Task 980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d8d9af40-e133-43f3-997d-455cc0a6c88a_isrForCrosstalkSources_1125923_31.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d8d9af40-e133-43f3-997d-455cc0a6c88a_isrForCrosstalkSources_1125923_31.stderr parsl.dataflow.dflow INFO: Task 966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53861af8556197862f0c3b2d8af31cc8 with result from task 966 parsl.dataflow.dflow DEBUG: Task 966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/ad9ded94-d000-47b0-afe1-8716d9c9ea89_isrForCrosstalkSources_1125941_31.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/ad9ded94-d000-47b0-afe1-8716d9c9ea89_isrForCrosstalkSources_1125941_31.stderr parsl.dataflow.dflow INFO: Task 979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e02013b7e56e32666fc10e5a4801668 with result from task 979 parsl.dataflow.dflow DEBUG: Task 979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/b36aadd1-a013-422f-8440-e584603496b3_isrForCrosstalkSources_1125923_29.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/b36aadd1-a013-422f-8440-e584603496b3_isrForCrosstalkSources_1125923_29.stderr parsl.dataflow.dflow INFO: Task 974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbd068bdcafcd3e971f351164f1a9db7 with result from task 974 parsl.dataflow.dflow DEBUG: Task 974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/4f2a4e81-3e03-400e-80e4-757c56faa7c0_isrForCrosstalkSources_1125913_62.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/4f2a4e81-3e03-400e-80e4-757c56faa7c0_isrForCrosstalkSources_1125913_62.stderr parsl.dataflow.dflow INFO: Task 977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bebfd4a38a2380775eb711e44e82138 with result from task 977 parsl.dataflow.dflow DEBUG: Task 977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/646010e5-a057-4bb2-8319-b711fede2aa7_isrForCrosstalkSources_1125888_10.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/646010e5-a057-4bb2-8319-b711fede2aa7_isrForCrosstalkSources_1125888_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:02 proc_lsst.multi:146 [INFO] found job 16896443 in provider escience_a40 proc_lsst.multi INFO: found job 16896443 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 971 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb182696430fc637ac20e7a7bd79a854 with result from task 971 parsl.dataflow.dflow DEBUG: Task 971 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/e841e1e0-d687-4956-a63e-f1aeb92dbd57_isrForCrosstalkSources_1125890_55.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/e841e1e0-d687-4956-a63e-f1aeb92dbd57_isrForCrosstalkSources_1125890_55.stderr parsl.dataflow.dflow INFO: Task 984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbb4efc2086b320580223753c139ec5c with result from task 984 parsl.dataflow.dflow DEBUG: Task 984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/a2322d84-e00b-4a40-be86-c661ea3bf636_isrForCrosstalkSources_1125893_37.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/a2322d84-e00b-4a40-be86-c661ea3bf636_isrForCrosstalkSources_1125893_37.stderr parsl.dataflow.dflow INFO: Task 985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89ed3fdfabd32fbdad7f30d31a871aa6 with result from task 985 parsl.dataflow.dflow DEBUG: Task 985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/ae49f39e-b649-44f8-a0b0-cded87704076_isrForCrosstalkSources_1125899_17.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/ae49f39e-b649-44f8-a0b0-cded87704076_isrForCrosstalkSources_1125899_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:02 proc_lsst.multi:146 [INFO] found job 16896444 in provider escience_a40 proc_lsst.multi INFO: found job 16896444 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6acccfeedd4e1c1bf9ec31ecbb0998d with result from task 987 parsl.dataflow.dflow DEBUG: Task 987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/409a2de7-ae10-4acf-a795-1f45e98ef1e2_isrForCrosstalkSources_1125901_45.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/409a2de7-ae10-4acf-a795-1f45e98ef1e2_isrForCrosstalkSources_1125901_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:03 proc_lsst.multi:146 [INFO] found job 16896445 in provider checkpoint proc_lsst.multi INFO: found job 16896445 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:03 proc_lsst.multi:146 [INFO] found job 16896446 in provider checkpoint proc_lsst.multi INFO: found job 16896446 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:04 proc_lsst.multi:146 [INFO] found job 16896447 in provider checkpoint proc_lsst.multi INFO: found job 16896447 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:04 proc_lsst.multi:146 [INFO] found job 16896448 in provider checkpoint proc_lsst.multi INFO: found job 16896448 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:05 proc_lsst.multi:146 [INFO] found job 16896449 in provider checkpoint proc_lsst.multi INFO: found job 16896449 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c902eabe98018dbb66db7b877b15d03e with result from task 986 parsl.dataflow.dflow DEBUG: Task 986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/67dbb3cb-8cb2-4417-b551-6a17a2bc0791_isrForCrosstalkSources_1125925_29.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/67dbb3cb-8cb2-4417-b551-6a17a2bc0791_isrForCrosstalkSources_1125925_29.stderr parsl.dataflow.dflow INFO: Task 1004 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 526f97221215ac5dce90916458beb7fb with result from task 1004 parsl.dataflow.dflow DEBUG: Task 1004 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/7daef095-6e44-49f5-9ec8-6c759c6cb8ba_isrForCrosstalkSources_1125925_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/7daef095-6e44-49f5-9ec8-6c759c6cb8ba_isrForCrosstalkSources_1125925_8.stderr parsl.dataflow.dflow INFO: Task 994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4918accc2769922e2be17b9aa77d222b with result from task 994 parsl.dataflow.dflow DEBUG: Task 994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6b17e937-22ca-490d-91e0-2c31a6f3dff2_isrForCrosstalkSources_1125909_29.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6b17e937-22ca-490d-91e0-2c31a6f3dff2_isrForCrosstalkSources_1125909_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:05 proc_lsst.multi:146 [INFO] found job 16896450 in provider checkpoint proc_lsst.multi INFO: found job 16896450 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99e1e8914bf4a049f66d55dfe494143f with result from task 993 parsl.dataflow.dflow DEBUG: Task 993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/77b17ec1-6dba-4403-8a5b-e6e674dce83f_isrForCrosstalkSources_1125889_15.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/77b17ec1-6dba-4403-8a5b-e6e674dce83f_isrForCrosstalkSources_1125889_15.stderr parsl.dataflow.dflow INFO: Task 1002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac861bfa495d6b1accaa393a6f4df31b with result from task 1002 parsl.dataflow.dflow DEBUG: Task 1002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3e42d99a-bc76-48b2-ae3c-29a72e4ba2de_isrForCrosstalkSources_1125941_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3e42d99a-bc76-48b2-ae3c-29a72e4ba2de_isrForCrosstalkSources_1125941_15.stderr parsl.dataflow.dflow INFO: Task 992 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cff76aca7428c5ce4ec0557f0b8132d6 with result from task 992 parsl.dataflow.dflow DEBUG: Task 992 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/8e1e64d8-eb46-4368-9cb7-16a663c80d10_isrForCrosstalkSources_1125925_30.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/8e1e64d8-eb46-4368-9cb7-16a663c80d10_isrForCrosstalkSources_1125925_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:06 proc_lsst.multi:146 [INFO] found job 16896451 in provider checkpoint proc_lsst.multi INFO: found job 16896451 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e034d70f44911521483c65ee28e6066c with result from task 998 parsl.dataflow.dflow DEBUG: Task 998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/fa5e7ae7-84c1-4a66-ab60-88feb57b550b_isrForCrosstalkSources_1125903_23.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/fa5e7ae7-84c1-4a66-ab60-88feb57b550b_isrForCrosstalkSources_1125903_23.stderr parsl.dataflow.dflow INFO: Task 996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a7f692cd0b57fa535a9b08c23a0f28b with result from task 996 parsl.dataflow.dflow DEBUG: Task 996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/ef608198-bd2e-4a69-ac96-38f3c34874f3_isrForCrosstalkSources_1125926_12.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/ef608198-bd2e-4a69-ac96-38f3c34874f3_isrForCrosstalkSources_1125926_12.stderr parsl.dataflow.dflow INFO: Task 1003 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbad35d0cb5796e7f007d5317d31292c with result from task 1003 parsl.dataflow.dflow DEBUG: Task 1003 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0c84a530-77ca-45d9-b689-1456b9bfd9b5_isrForCrosstalkSources_1125904_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0c84a530-77ca-45d9-b689-1456b9bfd9b5_isrForCrosstalkSources_1125904_22.stderr parsl.dataflow.dflow INFO: Task 1006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e60be2b288c64a446a9aa3f61478fa05 with result from task 1006 parsl.dataflow.dflow DEBUG: Task 1006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/7b75866f-37f3-47da-a8ab-3d3b0af8c99a_isrForCrosstalkSources_1125903_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/7b75866f-37f3-47da-a8ab-3d3b0af8c99a_isrForCrosstalkSources_1125903_15.stderr parsl.dataflow.dflow INFO: Task 1000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a615ac96a0ace5185cdc26b66f32a2eb with result from task 1000 parsl.dataflow.dflow DEBUG: Task 1000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/d86f9177-eb1f-4400-bda7-91e2280d6841_isrForCrosstalkSources_1125894_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/d86f9177-eb1f-4400-bda7-91e2280d6841_isrForCrosstalkSources_1125894_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:06 proc_lsst.multi:146 [INFO] found job 16896452 in provider checkpoint proc_lsst.multi INFO: found job 16896452 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a20d4dc3578641b0813e5fbe81203d1 with result from task 990 parsl.dataflow.dflow DEBUG: Task 990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/0b4a19e4-ee42-4cbf-9a92-c06497fb8274_isrForCrosstalkSources_1125894_30.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/0b4a19e4-ee42-4cbf-9a92-c06497fb8274_isrForCrosstalkSources_1125894_30.stderr parsl.dataflow.dflow INFO: Task 999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f75a9217d3a56f1325123ac5eb840edb with result from task 999 parsl.dataflow.dflow DEBUG: Task 999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/2640e34b-6276-4029-8372-8fa40a2fe72a_isrForCrosstalkSources_1125927_39.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/2640e34b-6276-4029-8372-8fa40a2fe72a_isrForCrosstalkSources_1125927_39.stderr parsl.dataflow.dflow INFO: Task 988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22fa3eb99d0e029335b8b88c0235482e with result from task 988 parsl.dataflow.dflow DEBUG: Task 988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/549df12a-a449-4e2d-9c04-5b1a97e956ee_isrForCrosstalkSources_1125940_60.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/549df12a-a449-4e2d-9c04-5b1a97e956ee_isrForCrosstalkSources_1125940_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:06 proc_lsst.multi:146 [INFO] found job 16896453 in provider checkpoint proc_lsst.multi INFO: found job 16896453 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6779b3c3c3a3b22f7a990a91e0057af2 with result from task 997 parsl.dataflow.dflow DEBUG: Task 997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/aeda80f8-b25d-4752-864c-3102c2ffadbc_isrForCrosstalkSources_1125930_46.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/aeda80f8-b25d-4752-864c-3102c2ffadbc_isrForCrosstalkSources_1125930_46.stderr parsl.dataflow.dflow INFO: Task 995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed6a0726852ed3205a3710ce7af3136b with result from task 995 parsl.dataflow.dflow DEBUG: Task 995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/fc1769ef-5f43-43d3-8003-e5abb37b29a0_isrForCrosstalkSources_1125938_13.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/fc1769ef-5f43-43d3-8003-e5abb37b29a0_isrForCrosstalkSources_1125938_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:07 proc_lsst.multi:146 [INFO] found job 16896454 in provider checkpoint proc_lsst.multi INFO: found job 16896454 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e738ce1d2dae73fe0bc14bc09dd54e5 with result from task 1001 parsl.dataflow.dflow DEBUG: Task 1001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/1f9fa766-e70c-4e79-a7a3-644e005194ad_isrForCrosstalkSources_1125903_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/1f9fa766-e70c-4e79-a7a3-644e005194ad_isrForCrosstalkSources_1125903_43.stderr parsl.dataflow.dflow INFO: Task 991 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 428d9df228a04d223f9802c9833aa164 with result from task 991 parsl.dataflow.dflow DEBUG: Task 991 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/42ed3622-fa9d-4ecd-9b90-c5c8580bd1fa_isrForCrosstalkSources_1125937_23.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/42ed3622-fa9d-4ecd-9b90-c5c8580bd1fa_isrForCrosstalkSources_1125937_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:07 proc_lsst.multi:146 [INFO] found job 16896455 in provider checkpoint proc_lsst.multi INFO: found job 16896455 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:08 proc_lsst.multi:146 [INFO] found job 16896456 in provider checkpoint proc_lsst.multi INFO: found job 16896456 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 989 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfb4a244bbb2c5cf0e84ecd55f776451 with result from task 989 parsl.dataflow.dflow DEBUG: Task 989 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/be2bb421-008c-4ca7-8acc-5601b12b4305_isrForCrosstalkSources_1125897_37.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/be2bb421-008c-4ca7-8acc-5601b12b4305_isrForCrosstalkSources_1125897_37.stderr parsl.dataflow.dflow INFO: Task 1028 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90753788753da8d373682f37efab18be with result from task 1028 parsl.dataflow.dflow DEBUG: Task 1028 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/31e4839e-63ff-4d87-94af-56a793d39c7d_isrForCrosstalkSources_1125891_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/31e4839e-63ff-4d87-94af-56a793d39c7d_isrForCrosstalkSources_1125891_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:08 proc_lsst.multi:146 [INFO] found job 16896457 in provider checkpoint proc_lsst.multi INFO: found job 16896457 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:08 proc_lsst.multi:146 [INFO] found job 16896458 in provider checkpoint proc_lsst.multi INFO: found job 16896458 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:08 proc_lsst.multi:146 [INFO] found job 16896459 in provider checkpoint proc_lsst.multi INFO: found job 16896459 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1037 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32f07d295499213e5d6f06194e98cdda with result from task 1037 parsl.dataflow.dflow DEBUG: Task 1037 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/24cdd39f-d1e4-4db5-8d57-835145341b1f_isrForCrosstalkSources_1125908_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/24cdd39f-d1e4-4db5-8d57-835145341b1f_isrForCrosstalkSources_1125908_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:09 proc_lsst.multi:146 [INFO] found job 16896460 in provider checkpoint proc_lsst.multi INFO: found job 16896460 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:09 proc_lsst.multi:146 [INFO] found job 16896461 in provider checkpoint proc_lsst.multi INFO: found job 16896461 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:09 proc_lsst.multi:146 [INFO] found job 16896462 in provider checkpoint proc_lsst.multi INFO: found job 16896462 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06d6e94b11468106f5e40c8a32f3ca79 with result from task 1005 parsl.dataflow.dflow DEBUG: Task 1005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/e7415e7f-ce47-44cc-8779-26a28d8801ef_isrForCrosstalkSources_1125914_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/e7415e7f-ce47-44cc-8779-26a28d8801ef_isrForCrosstalkSources_1125914_39.stderr parsl.dataflow.dflow INFO: Task 1040 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 471d41f8e8b3722ba8cfd0895e02bb5a with result from task 1040 parsl.dataflow.dflow DEBUG: Task 1040 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/765ea4b6-1d08-4e47-9f10-6a203b8d767e_isrForCrosstalkSources_1125930_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/765ea4b6-1d08-4e47-9f10-6a203b8d767e_isrForCrosstalkSources_1125930_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:09 proc_lsst.multi:146 [INFO] found job 16896463 in provider checkpoint proc_lsst.multi INFO: found job 16896463 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1044 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57faffa896964f38a03435a0330d3018 with result from task 1044 parsl.dataflow.dflow DEBUG: Task 1044 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/52b8ae9a-2181-4f6a-bea0-bb32592ded73_isrForCrosstalkSources_1125914_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/52b8ae9a-2181-4f6a-bea0-bb32592ded73_isrForCrosstalkSources_1125914_21.stderr parsl.dataflow.dflow INFO: Task 1038 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17b1d3ca581dde71857ed1e9d7496a8b with result from task 1038 parsl.dataflow.dflow DEBUG: Task 1038 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/90342246-beca-4e8a-9c08-5c1ca4d3945c_isrForCrosstalkSources_1125914_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/90342246-beca-4e8a-9c08-5c1ca4d3945c_isrForCrosstalkSources_1125914_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:10 proc_lsst.multi:146 [INFO] found job 16896464 in provider checkpoint proc_lsst.multi INFO: found job 16896464 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1043 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d36ce9f6aa5e2be11fc139fb296d0844 with result from task 1043 parsl.dataflow.dflow DEBUG: Task 1043 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/c608ac50-f7c7-4608-9474-34cb7c3e3865_isrForCrosstalkSources_1125898_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/c608ac50-f7c7-4608-9474-34cb7c3e3865_isrForCrosstalkSources_1125898_56.stderr parsl.dataflow.dflow INFO: Task 1049 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b912e3d063e65178f3430679186c665 with result from task 1049 parsl.dataflow.dflow DEBUG: Task 1049 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/05e20cac-57ac-4f58-b9ce-f30246b88e25_isrForCrosstalkSources_1125901_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/05e20cac-57ac-4f58-b9ce-f30246b88e25_isrForCrosstalkSources_1125901_22.stderr parsl.dataflow.dflow INFO: Task 1039 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f36f413645598385746511f66315d01f with result from task 1039 parsl.dataflow.dflow DEBUG: Task 1039 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/3226aba0-7e13-40ee-9087-fa70680b8154_isrForCrosstalkSources_1125900_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/3226aba0-7e13-40ee-9087-fa70680b8154_isrForCrosstalkSources_1125900_53.stderr parsl.dataflow.dflow INFO: Task 1047 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8260073bc9043b4a49729e8bcb232d6d with result from task 1047 parsl.dataflow.dflow DEBUG: Task 1047 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/39a6c9b4-39a6-431a-b892-447a02af3000_isrForCrosstalkSources_1125928_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/39a6c9b4-39a6-431a-b892-447a02af3000_isrForCrosstalkSources_1125928_14.stderr parsl.dataflow.dflow INFO: Task 1050 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 485abb6f70e8330ec46380284ed9c8f8 with result from task 1050 parsl.dataflow.dflow DEBUG: Task 1050 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/9cdab96c-d9fb-47b0-94d9-872babda8b42_isrForCrosstalkSources_1125899_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/9cdab96c-d9fb-47b0-94d9-872babda8b42_isrForCrosstalkSources_1125899_22.stderr parsl.dataflow.dflow INFO: Task 1031 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b52aeb3b7979c9df62d112bfa27a0408 with result from task 1031 parsl.dataflow.dflow DEBUG: Task 1031 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a1fdd4b4-5152-42d6-be3c-3d1eb1c891ee_isrForCrosstalkSources_1125900_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a1fdd4b4-5152-42d6-be3c-3d1eb1c891ee_isrForCrosstalkSources_1125900_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:11 proc_lsst.multi:146 [INFO] found job 16896466 in provider checkpoint proc_lsst.multi INFO: found job 16896466 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1045 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ec406b93354fd178124887715cc13a3 with result from task 1045 parsl.dataflow.dflow DEBUG: Task 1045 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/2dfd1ab9-9787-449c-836e-ff5db6a483f2_isrForCrosstalkSources_1125894_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/2dfd1ab9-9787-449c-836e-ff5db6a483f2_isrForCrosstalkSources_1125894_11.stderr parsl.dataflow.dflow INFO: Task 1048 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 899952601a4c222bb4f1a10e87cc97b0 with result from task 1048 parsl.dataflow.dflow DEBUG: Task 1048 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/2f64299a-2878-4ca1-978b-138521d143cf_isrForCrosstalkSources_1125922_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/2f64299a-2878-4ca1-978b-138521d143cf_isrForCrosstalkSources_1125922_61.stderr parsl.dataflow.dflow INFO: Task 1008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b313f21d592599220208cf9fb370326f with result from task 1008 parsl.dataflow.dflow DEBUG: Task 1008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/fe47c397-fa70-485c-9ed4-f3b4dc23e573_isrForCrosstalkSources_1125935_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/fe47c397-fa70-485c-9ed4-f3b4dc23e573_isrForCrosstalkSources_1125935_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:11 proc_lsst.multi:146 [INFO] found job 16896467 in provider checkpoint proc_lsst.multi INFO: found job 16896467 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1051 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3af9fd7ddc78631037eaf24f601eb1f2 with result from task 1051 parsl.dataflow.dflow DEBUG: Task 1051 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/0be30802-0544-4db0-ab13-76db154c3b64_isrForCrosstalkSources_1125908_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/0be30802-0544-4db0-ab13-76db154c3b64_isrForCrosstalkSources_1125908_9.stderr parsl.dataflow.dflow INFO: Task 1046 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 685171c77a7cd77880f4b990da5cfbdb with result from task 1046 parsl.dataflow.dflow DEBUG: Task 1046 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/c22b71a1-4649-4e0e-a5b9-a92cfb78b83e_isrForCrosstalkSources_1125934_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/c22b71a1-4649-4e0e-a5b9-a92cfb78b83e_isrForCrosstalkSources_1125934_36.stderr parsl.dataflow.dflow INFO: Task 1052 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0de9da140a64c94793cb41d5f279bbc8 with result from task 1052 parsl.dataflow.dflow DEBUG: Task 1052 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/71dcf546-367b-4827-856d-618e9b624797_isrForCrosstalkSources_1125904_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/71dcf546-367b-4827-856d-618e9b624797_isrForCrosstalkSources_1125904_1.stderr parsl.dataflow.dflow INFO: Task 1042 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0075b62e3df8f1969249bc77350623bd with result from task 1042 parsl.dataflow.dflow DEBUG: Task 1042 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/fe508941-b529-4924-95e9-f93cc5029f71_isrForCrosstalkSources_1125891_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/fe508941-b529-4924-95e9-f93cc5029f71_isrForCrosstalkSources_1125891_11.stderr parsl.dataflow.dflow INFO: Task 1034 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea2c6a2e547ad91ff01776e159ba7831 with result from task 1034 parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:11 proc_lsst.multi:146 [INFO] found job 16896468 in provider checkpoint proc_lsst.multi INFO: found job 16896468 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1034 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/e45b10ec-e384-4ef4-8b03-fe0eaafcd0f2_isrForCrosstalkSources_1125911_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/e45b10ec-e384-4ef4-8b03-fe0eaafcd0f2_isrForCrosstalkSources_1125911_24.stderr parsl.dataflow.dflow INFO: Task 1026 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7286a1788b6de484b9cf8e53be7ae150 with result from task 1026 parsl.dataflow.dflow DEBUG: Task 1026 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/d760895a-0058-4060-b2de-6102083d4587_isrForCrosstalkSources_1125912_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/d760895a-0058-4060-b2de-6102083d4587_isrForCrosstalkSources_1125912_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:12 proc_lsst.multi:146 [INFO] found job 16896469 in provider checkpoint proc_lsst.multi INFO: found job 16896469 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1033 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 613812839b72a985511b1a1b3a7ee6fc with result from task 1033 parsl.dataflow.dflow DEBUG: Task 1033 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/a4862c0d-ec44-44b0-82a1-7a5d3d0e6320_isrForCrosstalkSources_1125901_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/a4862c0d-ec44-44b0-82a1-7a5d3d0e6320_isrForCrosstalkSources_1125901_12.stderr parsl.dataflow.dflow INFO: Task 1032 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c66650b46c51b388c049b573ef9ae35c with result from task 1032 parsl.dataflow.dflow DEBUG: Task 1032 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/00619d1f-c4f0-47d2-b5dc-8dfe850cea37_isrForCrosstalkSources_1125943_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/00619d1f-c4f0-47d2-b5dc-8dfe850cea37_isrForCrosstalkSources_1125943_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:12 proc_lsst.multi:146 [INFO] found job 16896470 in provider checkpoint proc_lsst.multi INFO: found job 16896470 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e25a6db8705bdb062e3da3c645ceeac with result from task 1017 parsl.dataflow.dflow DEBUG: Task 1017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/80ad6aaa-5f33-4f76-97b6-d74d4c3dacdb_isrForCrosstalkSources_1125888_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/80ad6aaa-5f33-4f76-97b6-d74d4c3dacdb_isrForCrosstalkSources_1125888_48.stderr parsl.dataflow.dflow INFO: Task 1053 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f69e79d0da647c97c605bc1c0ac58ddf with result from task 1053 parsl.dataflow.dflow DEBUG: Task 1053 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/a63c9ee1-9d08-48be-ade3-230a7485e2d2_isrForCrosstalkSources_1125896_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/a63c9ee1-9d08-48be-ade3-230a7485e2d2_isrForCrosstalkSources_1125896_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:12 proc_lsst.multi:146 [INFO] found job 16896471 in provider checkpoint proc_lsst.multi INFO: found job 16896471 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1041 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 405f078a4df3d52dd39a1cbf8bbf5779 with result from task 1041 parsl.dataflow.dflow DEBUG: Task 1041 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/3d1a00ab-8800-4792-aab2-df9dc8a22c35_isrForCrosstalkSources_1125906_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/3d1a00ab-8800-4792-aab2-df9dc8a22c35_isrForCrosstalkSources_1125906_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:12 proc_lsst.multi:146 [INFO] found job 16896472 in provider checkpoint proc_lsst.multi INFO: found job 16896472 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39b42277c84e2665d1d2878be64323ce with result from task 1007 parsl.dataflow.dflow DEBUG: Task 1007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/fec609fc-4b32-4df8-9eaf-13a14a7fd88e_isrForCrosstalkSources_1125888_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/fec609fc-4b32-4df8-9eaf-13a14a7fd88e_isrForCrosstalkSources_1125888_52.stderr parsl.dataflow.dflow INFO: Task 1029 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f413fe49e47e68c26d29f617eb7402d with result from task 1029 parsl.dataflow.dflow DEBUG: Task 1029 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/a4c17cdd-abe5-41bb-af05-2492db2afac1_isrForCrosstalkSources_1125916_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/a4c17cdd-abe5-41bb-af05-2492db2afac1_isrForCrosstalkSources_1125916_36.stderr parsl.dataflow.dflow INFO: Task 1027 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5effd61836a01c8531e045fd230b1846 with result from task 1027 parsl.dataflow.dflow DEBUG: Task 1027 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/266de6d8-4567-4f02-bfbb-a16b1c61b0a5_isrForCrosstalkSources_1125944_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/266de6d8-4567-4f02-bfbb-a16b1c61b0a5_isrForCrosstalkSources_1125944_36.stderr parsl.dataflow.dflow INFO: Task 1035 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d02054b28933ec602c7f0a840f6c9ca with result from task 1035 parsl.dataflow.dflow DEBUG: Task 1035 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9d571f5e-0b55-4432-89e2-d3fa77210717_isrForCrosstalkSources_1125934_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9d571f5e-0b55-4432-89e2-d3fa77210717_isrForCrosstalkSources_1125934_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:12 proc_lsst.multi:146 [INFO] found job 16896473 in provider checkpoint proc_lsst.multi INFO: found job 16896473 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1030 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94147e8fc5e0f5522a56e5f24d23eb79 with result from task 1030 parsl.dataflow.dflow DEBUG: Task 1030 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/31059be7-e114-4971-80cd-35a255d0dd32_isrForCrosstalkSources_1125936_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/31059be7-e114-4971-80cd-35a255d0dd32_isrForCrosstalkSources_1125936_61.stderr parsl.dataflow.dflow INFO: Task 1036 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38276af9041f2f9e7e7dd2a4a8ea403b with result from task 1036 parsl.dataflow.dflow DEBUG: Task 1036 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/c3356a0f-c7f9-474f-99f5-164705ce4e27_isrForCrosstalkSources_1125919_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/c3356a0f-c7f9-474f-99f5-164705ce4e27_isrForCrosstalkSources_1125919_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:13 proc_lsst.multi:146 [INFO] found job 16896474 in provider checkpoint proc_lsst.multi INFO: found job 16896474 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896449 PD 16896450 PD 16896451 PD 16896452 PD 16896453 PD 16896454 PD 16896455 PD 16896456 PD 16896457 PD 16896458 PD 16896459 PD 16896460 PD 16896461 PD 16896462 PD 16896463 PD 16896464 PD 16896466 PD 16896467 PD 16896468 PD 16896469 PD 16896470 PD 16896471 PD 16896472 PD 16896473 PD 16896474 PD 16896475 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896477 PD 16896476 PD 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:14 proc_lsst.multi:146 [INFO] found job 16896475 in provider checkpoint proc_lsst.multi INFO: found job 16896475 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:16 proc_lsst.multi:146 [INFO] found job 16896476 in provider checkpoint proc_lsst.multi INFO: found job 16896476 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1057 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 266d4e0df286633f21b8e20565c133d7 with result from task 1057 parsl.dataflow.dflow DEBUG: Task 1057 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/e1baa4cb-2b3b-4701-9362-180e362d70ae_isrForCrosstalkSources_1125889_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/e1baa4cb-2b3b-4701-9362-180e362d70ae_isrForCrosstalkSources_1125889_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:16 proc_lsst.multi:146 [INFO] found job 16896477 in provider checkpoint proc_lsst.multi INFO: found job 16896477 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1064 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e24ba14724fae5a4f0ac276106555a39 with result from task 1064 parsl.dataflow.dflow DEBUG: Task 1064 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/52874518-ba3d-4752-a17a-c09e41b10e57_isrForCrosstalkSources_1125936_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/52874518-ba3d-4752-a17a-c09e41b10e57_isrForCrosstalkSources_1125936_17.stderr parsl.dataflow.dflow INFO: Task 1056 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14ca390ed141a167eae79831aa502da5 with result from task 1056 parsl.dataflow.dflow DEBUG: Task 1056 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/3b75714f-7442-4ceb-b63e-00631e68a188_isrForCrosstalkSources_1125903_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/3b75714f-7442-4ceb-b63e-00631e68a188_isrForCrosstalkSources_1125903_22.stderr parsl.dataflow.dflow INFO: Task 1073 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 834873a60fce5d326f05ababc231525c with result from task 1073 parsl.dataflow.dflow DEBUG: Task 1073 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/2fd19f65-ed17-481c-8305-e394105c681d_isrForCrosstalkSources_1125934_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/2fd19f65-ed17-481c-8305-e394105c681d_isrForCrosstalkSources_1125934_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:16 proc_lsst.multi:146 [INFO] found job 16896478 in provider checkpoint proc_lsst.multi INFO: found job 16896478 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:17 proc_lsst.multi:146 [INFO] found job 16896479 in provider checkpoint proc_lsst.multi INFO: found job 16896479 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1060 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae61cbe4a6b934f7729f179528a5bb56 with result from task 1060 parsl.dataflow.dflow DEBUG: Task 1060 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/65d8aa89-daa0-4e12-aa84-6721e904984c_isrForCrosstalkSources_1125903_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/65d8aa89-daa0-4e12-aa84-6721e904984c_isrForCrosstalkSources_1125903_2.stderr parsl.dataflow.dflow INFO: Task 1067 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdd7a7010a40e265a02b99922f69907c with result from task 1067 parsl.dataflow.dflow DEBUG: Task 1067 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/9415764e-8a48-4c40-95a0-95f549ab86d8_isrForCrosstalkSources_1125946_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/9415764e-8a48-4c40-95a0-95f549ab86d8_isrForCrosstalkSources_1125946_4.stderr parsl.dataflow.dflow INFO: Task 1071 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d926c509ce224fa6fa4bf6b8d6f99d4 with result from task 1071 parsl.dataflow.dflow DEBUG: Task 1071 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/61dc49f3-87d8-4db5-ab17-1e82cf299b46_isrForCrosstalkSources_1125947_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/61dc49f3-87d8-4db5-ab17-1e82cf299b46_isrForCrosstalkSources_1125947_25.stderr parsl.dataflow.dflow INFO: Task 1074 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1053594514f0d0ace1b9552ff2f3ed22 with result from task 1074 parsl.dataflow.dflow DEBUG: Task 1074 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/d38623c8-99ca-49e7-9da9-a4962b28f99d_isrForCrosstalkSources_1125927_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/d38623c8-99ca-49e7-9da9-a4962b28f99d_isrForCrosstalkSources_1125927_62.stderr parsl.dataflow.dflow INFO: Task 1066 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 066463ca4f694c2f62449f69c5c734b5 with result from task 1066 parsl.dataflow.dflow DEBUG: Task 1066 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/b2fdf050-07f9-492d-ac01-0db9a272df68_isrForCrosstalkSources_1125918_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/b2fdf050-07f9-492d-ac01-0db9a272df68_isrForCrosstalkSources_1125918_35.stderr parsl.dataflow.dflow INFO: Task 1065 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d380d70681697f48306406a69900609e with result from task 1065 parsl.dataflow.dflow DEBUG: Task 1065 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/726fe89b-52a8-4329-a9c9-1a5c0ca0ef98_isrForCrosstalkSources_1125912_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/726fe89b-52a8-4329-a9c9-1a5c0ca0ef98_isrForCrosstalkSources_1125912_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:17 proc_lsst.multi:146 [INFO] found job 16896480 in provider checkpoint proc_lsst.multi INFO: found job 16896480 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1054 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8ed4f294f12a1a4e880b77d603188d9 with result from task 1054 parsl.dataflow.dflow DEBUG: Task 1054 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/ba7bd594-f745-4bf7-a1f3-ffe1367c4907_isrForCrosstalkSources_1125898_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/ba7bd594-f745-4bf7-a1f3-ffe1367c4907_isrForCrosstalkSources_1125898_25.stderr parsl.dataflow.dflow INFO: Task 1059 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ff335a06c32b06f5f1f4edd19583433 with result from task 1059 parsl.dataflow.dflow DEBUG: Task 1059 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/d86c36b0-b986-4975-94a2-b54d921b3d17_isrForCrosstalkSources_1125899_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/d86c36b0-b986-4975-94a2-b54d921b3d17_isrForCrosstalkSources_1125899_47.stderr parsl.dataflow.dflow INFO: Task 1076 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0800d10d46bffcb74781f8f7556c9da with result from task 1076 parsl.dataflow.dflow DEBUG: Task 1076 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/a681c070-3292-4c67-82aa-b421e235e5a9_isrForCrosstalkSources_1125892_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/a681c070-3292-4c67-82aa-b421e235e5a9_isrForCrosstalkSources_1125892_45.stderr parsl.dataflow.dflow INFO: Task 1058 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1044cb01bf7e7c309c3fd75009f21b4f with result from task 1058 parsl.dataflow.dflow DEBUG: Task 1058 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/d8af5b33-59eb-4952-a1b2-4d55bc82de2c_isrForCrosstalkSources_1125936_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/d8af5b33-59eb-4952-a1b2-4d55bc82de2c_isrForCrosstalkSources_1125936_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:18 proc_lsst.multi:146 [INFO] found job 16896481 in provider checkpoint proc_lsst.multi INFO: found job 16896481 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1070 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4180c204242866f85ef053479514cbe with result from task 1070 parsl.dataflow.dflow DEBUG: Task 1070 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/c6fb3f64-dae7-4f3a-91b0-1fd468d2ceff_isrForCrosstalkSources_1125894_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/c6fb3f64-dae7-4f3a-91b0-1fd468d2ceff_isrForCrosstalkSources_1125894_12.stderr parsl.dataflow.dflow INFO: Task 1068 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d243935427de07d83313ac1fbd85f2d2 with result from task 1068 parsl.dataflow.dflow DEBUG: Task 1068 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/271bfc21-930c-4916-867c-6221644ca047_isrForCrosstalkSources_1125930_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/271bfc21-930c-4916-867c-6221644ca047_isrForCrosstalkSources_1125930_48.stderr parsl.dataflow.dflow INFO: Task 1069 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db35698dc76a89ead0036bbeef0af830 with result from task 1069 parsl.dataflow.dflow DEBUG: Task 1069 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/0ff861f6-fc01-416e-8ec5-82940fe21d8c_isrForCrosstalkSources_1125947_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/0ff861f6-fc01-416e-8ec5-82940fe21d8c_isrForCrosstalkSources_1125947_53.stderr parsl.dataflow.dflow INFO: Task 1063 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03abd6cda47afb7a642f96b0d5ec1b88 with result from task 1063 parsl.dataflow.dflow DEBUG: Task 1063 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/0eef2f80-34da-43ea-98f7-13669ed4a9a7_isrForCrosstalkSources_1125898_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/0eef2f80-34da-43ea-98f7-13669ed4a9a7_isrForCrosstalkSources_1125898_47.stderr parsl.dataflow.dflow INFO: Task 1055 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ae14e7cfd7e7cb57f9794a22b4cb851 with result from task 1055 parsl.dataflow.dflow DEBUG: Task 1055 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/d7f57580-01a9-450e-8bc9-7c8a95e1e048_isrForCrosstalkSources_1125914_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/d7f57580-01a9-450e-8bc9-7c8a95e1e048_isrForCrosstalkSources_1125914_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 1062 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26157f07a8de2135ab04a752187bbbdd with result from task 1062 parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:18 proc_lsst.multi:146 [INFO] found job 16896482 in provider checkpoint proc_lsst.multi INFO: found job 16896482 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1062 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/7999f8e7-156d-414c-817c-f40001981e7a_isrForCrosstalkSources_1125945_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/7999f8e7-156d-414c-817c-f40001981e7a_isrForCrosstalkSources_1125945_41.stderr parsl.dataflow.dflow INFO: Task 1061 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc2db30ff24c952de72f8f08593ef76a with result from task 1061 parsl.dataflow.dflow DEBUG: Task 1061 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/da847968-87b1-4f8f-9146-04bd80950c48_isrForCrosstalkSources_1125933_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/da847968-87b1-4f8f-9146-04bd80950c48_isrForCrosstalkSources_1125933_61.stderr parsl.dataflow.dflow INFO: Task 1072 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3004730c6e0fa113004e496a8c05cd0e with result from task 1072 parsl.dataflow.dflow DEBUG: Task 1072 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/fd5117ad-2b16-40bf-a1a2-e573cd8462cf_isrForCrosstalkSources_1125928_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/fd5117ad-2b16-40bf-a1a2-e573cd8462cf_isrForCrosstalkSources_1125928_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:19 proc_lsst.multi:146 [INFO] found job 16896483 in provider checkpoint proc_lsst.multi INFO: found job 16896483 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:19 proc_lsst.multi:146 [INFO] found job 16896484 in provider checkpoint proc_lsst.multi INFO: found job 16896484 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1085 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 923964ab20b6c6c9faf48c3a9fd787bc with result from task 1085 parsl.dataflow.dflow DEBUG: Task 1085 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/c1298692-0bd2-48bc-b454-d380e715c311_isrForCrosstalkSources_1125928_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/c1298692-0bd2-48bc-b454-d380e715c311_isrForCrosstalkSources_1125928_27.stderr parsl.dataflow.dflow INFO: Task 1078 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6aa420b4e7749d0b3e22cfd0e1acff72 with result from task 1078 parsl.dataflow.dflow DEBUG: Task 1078 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/7b98e072-eceb-458f-b9db-d037d135fc7e_isrForCrosstalkSources_1125897_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/7b98e072-eceb-458f-b9db-d037d135fc7e_isrForCrosstalkSources_1125897_56.stderr parsl.dataflow.dflow INFO: Task 1082 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f852bbe02c193508cf8101300360a612 with result from task 1082 parsl.dataflow.dflow DEBUG: Task 1082 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/f99a9ee3-7ae2-4cd5-ba0f-47a815a52235_isrForCrosstalkSources_1125928_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/f99a9ee3-7ae2-4cd5-ba0f-47a815a52235_isrForCrosstalkSources_1125928_46.stderr parsl.dataflow.dflow INFO: Task 1081 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f4cf70c635896b4fcb62ddb802248fa with result from task 1081 parsl.dataflow.dflow DEBUG: Task 1081 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/9f134e5a-ab64-4dee-b30d-d2cb40053981_isrForCrosstalkSources_1125924_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/9f134e5a-ab64-4dee-b30d-d2cb40053981_isrForCrosstalkSources_1125924_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:20 proc_lsst.multi:146 [INFO] found job 16896485 in provider checkpoint proc_lsst.multi INFO: found job 16896485 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1088 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52320a385d9dc329db6064f6d4d2fdac with result from task 1088 parsl.dataflow.dflow DEBUG: Task 1088 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/0f7134ce-000a-46d9-9424-3774e96cf259_isrForCrosstalkSources_1125928_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/0f7134ce-000a-46d9-9424-3774e96cf259_isrForCrosstalkSources_1125928_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:20 proc_lsst.multi:146 [INFO] found job 16896486 in provider checkpoint proc_lsst.multi INFO: found job 16896486 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1080 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fc3f2042559a5e5736da49d8fc898ad with result from task 1080 parsl.dataflow.dflow DEBUG: Task 1080 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/90080d39-7049-4768-a0c2-db2cdc742db4_isrForCrosstalkSources_1125920_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/90080d39-7049-4768-a0c2-db2cdc742db4_isrForCrosstalkSources_1125920_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:21 proc_lsst.multi:146 [INFO] found job 16896487 in provider checkpoint proc_lsst.multi INFO: found job 16896487 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1090 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09e98d770058e06916a0f3ad40209ab1 with result from task 1090 parsl.dataflow.dflow DEBUG: Task 1090 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/9ee296eb-7a42-4528-a1e2-c5536a9ddfbf_isrForCrosstalkSources_1125942_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/9ee296eb-7a42-4528-a1e2-c5536a9ddfbf_isrForCrosstalkSources_1125942_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:21 proc_lsst.multi:146 [INFO] found job 16896488 in provider checkpoint proc_lsst.multi INFO: found job 16896488 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1075 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b136c3c1f36af802f7051dfd45a28e20 with result from task 1075 parsl.dataflow.dflow DEBUG: Task 1075 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/3116fd6d-0cce-493e-8453-b884dc2a9f0e_isrForCrosstalkSources_1125919_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/3116fd6d-0cce-493e-8453-b884dc2a9f0e_isrForCrosstalkSources_1125919_35.stderr parsl.dataflow.dflow INFO: Task 1097 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dd3bdecb317f23ee08ab3f4b4e5fb06 with result from task 1097 parsl.dataflow.dflow DEBUG: Task 1097 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/973ebdb1-2233-463a-9004-916d391d3747_isrForCrosstalkSources_1125904_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/973ebdb1-2233-463a-9004-916d391d3747_isrForCrosstalkSources_1125904_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:22 proc_lsst.multi:146 [INFO] found job 16896489 in provider checkpoint proc_lsst.multi INFO: found job 16896489 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1089 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78cf51e67ab4f7807551e7e403f53d1f with result from task 1089 parsl.dataflow.dflow DEBUG: Task 1089 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/af78e220-d447-471f-a044-6c1e73937002_isrForCrosstalkSources_1125939_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/af78e220-d447-471f-a044-6c1e73937002_isrForCrosstalkSources_1125939_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:22 proc_lsst.multi:146 [INFO] found job 16896490 in provider checkpoint proc_lsst.multi INFO: found job 16896490 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:23 proc_lsst.multi:146 [INFO] found job 16896491 in provider checkpoint proc_lsst.multi INFO: found job 16896491 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1095 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02a9632cc2e9924bb08072176e054f7b with result from task 1095 parsl.dataflow.dflow DEBUG: Task 1095 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/4c8749d6-b0e6-4650-9963-c6269563b6a3_isrForCrosstalkSources_1125944_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/4c8749d6-b0e6-4650-9963-c6269563b6a3_isrForCrosstalkSources_1125944_39.stderr parsl.dataflow.dflow INFO: Task 1093 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 344c51f96587c1fefb499bc808f8b1a5 with result from task 1093 parsl.dataflow.dflow DEBUG: Task 1093 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/07ac3019-2b72-445e-adce-3a108b816641_isrForCrosstalkSources_1125941_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/07ac3019-2b72-445e-adce-3a108b816641_isrForCrosstalkSources_1125941_4.stderr parsl.dataflow.dflow INFO: Task 1077 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef394c09d6ffae6e4f6ad3a14915a7c6 with result from task 1077 parsl.dataflow.dflow DEBUG: Task 1077 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a9854239-66c7-46ea-9dcc-5b74b6cb1196_isrForCrosstalkSources_1125921_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a9854239-66c7-46ea-9dcc-5b74b6cb1196_isrForCrosstalkSources_1125921_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:23 proc_lsst.multi:146 [INFO] found job 16896492 in provider checkpoint proc_lsst.multi INFO: found job 16896492 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1086 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f3b697db21dc78a8de354bc732e3759 with result from task 1086 parsl.dataflow.dflow DEBUG: Task 1086 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/c735aa07-1259-4f54-8472-28e0cbf2ce51_isrForCrosstalkSources_1125924_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/c735aa07-1259-4f54-8472-28e0cbf2ce51_isrForCrosstalkSources_1125924_13.stderr parsl.dataflow.dflow INFO: Task 1096 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 591f98b82ebe9395bde6ce7f734c5de9 with result from task 1096 parsl.dataflow.dflow DEBUG: Task 1096 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/eea7a7f4-f814-4a31-9b56-b7ad91c8f776_isrForCrosstalkSources_1125921_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/eea7a7f4-f814-4a31-9b56-b7ad91c8f776_isrForCrosstalkSources_1125921_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:23 proc_lsst.multi:146 [INFO] found job 16896493 in provider checkpoint proc_lsst.multi INFO: found job 16896493 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1079 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 172abe287accd7aae1c42e30ecc39f16 with result from task 1079 parsl.dataflow.dflow DEBUG: Task 1079 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/6a1f1274-db59-4b0a-a64e-b14e7b1d70ac_isrForCrosstalkSources_1125893_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/6a1f1274-db59-4b0a-a64e-b14e7b1d70ac_isrForCrosstalkSources_1125893_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:23 proc_lsst.multi:146 [INFO] found job 16896494 in provider checkpoint proc_lsst.multi INFO: found job 16896494 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1084 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 044bf44dd6f24cc6b8607bc7c802348b with result from task 1084 parsl.dataflow.dflow DEBUG: Task 1084 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/778bd707-4a8d-44b4-b667-45cbf2726360_isrForCrosstalkSources_1125921_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/778bd707-4a8d-44b4-b667-45cbf2726360_isrForCrosstalkSources_1125921_21.stderr parsl.dataflow.dflow INFO: Task 1104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0dd857224e67763bada7f5a35d7ad4c8 with result from task 1104 parsl.dataflow.dflow DEBUG: Task 1104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/d564c2cf-ad62-48be-bd75-71b038c8e329_isrForCrosstalkSources_1125900_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/d564c2cf-ad62-48be-bd75-71b038c8e329_isrForCrosstalkSources_1125900_45.stderr parsl.dataflow.dflow INFO: Task 1100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 375e9e6028e37458b0f34dd7a361d10f with result from task 1100 parsl.dataflow.dflow DEBUG: Task 1100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/fac629ea-a6a3-476a-ae2a-c7ddaf08edb5_isrForCrosstalkSources_1125895_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/fac629ea-a6a3-476a-ae2a-c7ddaf08edb5_isrForCrosstalkSources_1125895_45.stderr parsl.dataflow.dflow INFO: Task 1091 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35211293074c478ee36528331baef72f with result from task 1091 parsl.dataflow.dflow DEBUG: Task 1091 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/98b57a0e-49f0-402c-89aa-f0851f4447a6_isrForCrosstalkSources_1125906_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/98b57a0e-49f0-402c-89aa-f0851f4447a6_isrForCrosstalkSources_1125906_16.stderr parsl.dataflow.dflow INFO: Task 1087 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 419bd8bf458c5d900479dc013c710466 with result from task 1087 parsl.dataflow.dflow DEBUG: Task 1087 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/4dcc754d-e0d9-43cf-8d10-85cf4694ca97_isrForCrosstalkSources_1125943_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/4dcc754d-e0d9-43cf-8d10-85cf4694ca97_isrForCrosstalkSources_1125943_11.stderr parsl.dataflow.dflow INFO: Task 1094 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d1f3a50e6a47087836e8548563edb10 with result from task 1094 parsl.dataflow.dflow DEBUG: Task 1094 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a29f6450-1edc-4a88-9638-19317a3e94cc_isrForCrosstalkSources_1125945_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a29f6450-1edc-4a88-9638-19317a3e94cc_isrForCrosstalkSources_1125945_16.stderr parsl.dataflow.dflow INFO: Task 1101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 491f769b46d76b2a72daa382af4288a3 with result from task 1101 parsl.dataflow.dflow DEBUG: Task 1101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/02da22bb-976e-4992-bc5d-62918018fcb5_isrForCrosstalkSources_1125919_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/02da22bb-976e-4992-bc5d-62918018fcb5_isrForCrosstalkSources_1125919_2.stderr parsl.dataflow.dflow INFO: Task 1098 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95dcb6c0ffbdfde7ea68dc0a466c95b7 with result from task 1098 parsl.dataflow.dflow DEBUG: Task 1098 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/f2c334e1-df4c-4483-87e9-b511413644ee_isrForCrosstalkSources_1125925_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/f2c334e1-df4c-4483-87e9-b511413644ee_isrForCrosstalkSources_1125925_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:24 proc_lsst.multi:146 [INFO] found job 16896495 in provider checkpoint proc_lsst.multi INFO: found job 16896495 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1083 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b17a08c238901724f20d31bd8c3edad with result from task 1083 parsl.dataflow.dflow DEBUG: Task 1083 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/be5a26ff-d00b-4210-b096-799ffec0986d_isrForCrosstalkSources_1125935_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/be5a26ff-d00b-4210-b096-799ffec0986d_isrForCrosstalkSources_1125935_53.stderr parsl.dataflow.dflow INFO: Task 1103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5443cf54874fc437cbd925777f6fe8c4 with result from task 1103 parsl.dataflow.dflow DEBUG: Task 1103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/0c25c76a-9d22-4cfb-b77b-f76118fa2172_isrForCrosstalkSources_1125934_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/0c25c76a-9d22-4cfb-b77b-f76118fa2172_isrForCrosstalkSources_1125934_54.stderr parsl.dataflow.dflow INFO: Task 1092 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbe4e9b6e5d98a4916e086461e996194 with result from task 1092 parsl.dataflow.dflow DEBUG: Task 1092 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/75fd488c-2437-4778-96d2-00d8ec506d4e_isrForCrosstalkSources_1125944_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/75fd488c-2437-4778-96d2-00d8ec506d4e_isrForCrosstalkSources_1125944_60.stderr parsl.dataflow.dflow INFO: Task 1108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d7dd0f880d2c160d241119595bccbac with result from task 1108 parsl.dataflow.dflow DEBUG: Task 1108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/535fa8e8-e801-4e0e-ad14-bf8b9a3a18d8_isrForCrosstalkSources_1125940_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/535fa8e8-e801-4e0e-ad14-bf8b9a3a18d8_isrForCrosstalkSources_1125940_14.stderr parsl.dataflow.dflow INFO: Task 1102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b8d52ef69524f601ce77f84132f0b15 with result from task 1102 parsl.dataflow.dflow DEBUG: Task 1102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2a83470b-c7ab-4c01-9821-417c7fa86938_isrForCrosstalkSources_1125898_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2a83470b-c7ab-4c01-9821-417c7fa86938_isrForCrosstalkSources_1125898_42.stderr parsl.dataflow.dflow INFO: Task 1112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d4893f2d6c09ab3a1722ae15423c0f9 with result from task 1112 parsl.dataflow.dflow DEBUG: Task 1112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/4cd57217-d65b-49a1-8fa3-f36181d0dc4c_isrForCrosstalkSources_1125895_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/4cd57217-d65b-49a1-8fa3-f36181d0dc4c_isrForCrosstalkSources_1125895_60.stderr parsl.dataflow.dflow INFO: Task 1117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6307a69ed81f5483bb52de6328628f0 with result from task 1117 parsl.dataflow.dflow DEBUG: Task 1117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/6e7e5217-d9de-41bc-935c-aa8414487b7a_isrForCrosstalkSources_1125943_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/6e7e5217-d9de-41bc-935c-aa8414487b7a_isrForCrosstalkSources_1125943_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:27 proc_lsst.multi:146 [INFO] found job 16896496 in provider checkpoint proc_lsst.multi INFO: found job 16896496 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 500b3e302decdd16b1f594184cff96f1 with result from task 1107 parsl.dataflow.dflow DEBUG: Task 1107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/7f4ddf15-1349-4b3b-a0e6-9c71078eef61_isrForCrosstalkSources_1125918_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/7f4ddf15-1349-4b3b-a0e6-9c71078eef61_isrForCrosstalkSources_1125918_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:27 proc_lsst.multi:146 [INFO] found job 16896497 in provider checkpoint proc_lsst.multi INFO: found job 16896497 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:27 proc_lsst.multi:146 [INFO] found job 16896498 in provider checkpoint proc_lsst.multi INFO: found job 16896498 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1099 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7323b04c9f599e91870d6cf57443c763 with result from task 1099 parsl.dataflow.dflow DEBUG: Task 1099 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/2d7abb00-11c3-486b-b206-6d0819e185e0_isrForCrosstalkSources_1125901_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/2d7abb00-11c3-486b-b206-6d0819e185e0_isrForCrosstalkSources_1125901_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:28 proc_lsst.multi:146 [INFO] found job 16896499 in provider checkpoint proc_lsst.multi INFO: found job 16896499 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c94d56b1f2b05a7cd8c65de4b30faaf with result from task 1009 parsl.dataflow.dflow DEBUG: Task 1009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/650676a5-4700-454f-9f9c-0e4901117e05_isrForCrosstalkSources_1125936_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/650676a5-4700-454f-9f9c-0e4901117e05_isrForCrosstalkSources_1125936_4.stderr parsl.dataflow.dflow INFO: Task 1016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 973198b2fd74aa7cfdc1c23d028b085d with result from task 1016 parsl.dataflow.dflow DEBUG: Task 1016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/aa5ffa79-38dd-4133-9d19-d0e66ad3bee4_isrForCrosstalkSources_1125932_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/aa5ffa79-38dd-4133-9d19-d0e66ad3bee4_isrForCrosstalkSources_1125932_10.stderr parsl.dataflow.dflow INFO: Task 1014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba2a19ca5f2c473b5b1c15c200f21a1c with result from task 1014 parsl.dataflow.dflow DEBUG: Task 1014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/0bf3e32a-2940-4bb2-ae61-363b6ac0ab0e_isrForCrosstalkSources_1125940_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/0bf3e32a-2940-4bb2-ae61-363b6ac0ab0e_isrForCrosstalkSources_1125940_54.stderr parsl.dataflow.dflow INFO: Task 1015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c28cb287d2aad4ea23de4ca6cb69a355 with result from task 1015 parsl.dataflow.dflow DEBUG: Task 1015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/dea87cae-dbcf-4d98-9673-1d1656ffdbac_isrForCrosstalkSources_1125931_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/dea87cae-dbcf-4d98-9673-1d1656ffdbac_isrForCrosstalkSources_1125931_55.stderr parsl.dataflow.dflow INFO: Task 1010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e3b2601d544b33d4659dfe59d057f3a with result from task 1010 parsl.dataflow.dflow DEBUG: Task 1010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/86ddf116-ffd3-4795-8834-4d4cb7aeabd1_isrForCrosstalkSources_1125924_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/86ddf116-ffd3-4795-8834-4d4cb7aeabd1_isrForCrosstalkSources_1125924_16.stderr parsl.dataflow.dflow INFO: Task 1012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ed93bced951a192ff1c8ba089b5e38b with result from task 1012 parsl.dataflow.dflow DEBUG: Task 1012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/1b1192ed-58d0-470c-886d-a04ad25ab038_isrForCrosstalkSources_1125910_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/1b1192ed-58d0-470c-886d-a04ad25ab038_isrForCrosstalkSources_1125910_42.stderr parsl.dataflow.dflow INFO: Task 1011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e93539a1541f0a0dbc62c0ccea82fd0 with result from task 1011 parsl.dataflow.dflow DEBUG: Task 1011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/ca3c059c-b787-4864-bc1f-219b17c4a54a_isrForCrosstalkSources_1125929_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/ca3c059c-b787-4864-bc1f-219b17c4a54a_isrForCrosstalkSources_1125929_12.stderr parsl.dataflow.dflow INFO: Task 1013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f13787ae3c6e9cae93d984cb2ec13495 with result from task 1013 parsl.dataflow.dflow DEBUG: Task 1013 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/15591753-1ab5-4809-abb6-1b5ca0768854_isrForCrosstalkSources_1125941_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/15591753-1ab5-4809-abb6-1b5ca0768854_isrForCrosstalkSources_1125941_53.stderr parsl.dataflow.dflow INFO: Task 1116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e5e19b3854d6e433590472d2f6cf095 with result from task 1116 parsl.dataflow.dflow DEBUG: Task 1116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/5b4e42e2-58c6-4f39-acf6-929077d40f92_isrForCrosstalkSources_1125925_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/5b4e42e2-58c6-4f39-acf6-929077d40f92_isrForCrosstalkSources_1125925_47.stderr parsl.dataflow.dflow INFO: Task 1119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f4cd5b2f8ab2831da5bd1af642dc674 with result from task 1119 parsl.dataflow.dflow DEBUG: Task 1119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/35286ee2-540f-4bc4-a855-ea8032d78b4c_isrForCrosstalkSources_1125924_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/35286ee2-540f-4bc4-a855-ea8032d78b4c_isrForCrosstalkSources_1125924_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:28 proc_lsst.multi:146 [INFO] found job 16896500 in provider checkpoint proc_lsst.multi INFO: found job 16896500 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7bbb2b779a3b2ceb9c7fd7b4960c42b with result from task 1114 parsl.dataflow.dflow DEBUG: Task 1114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/882cabcb-58e7-40a5-b906-7fe7f47a8425_isrForCrosstalkSources_1125897_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/882cabcb-58e7-40a5-b906-7fe7f47a8425_isrForCrosstalkSources_1125897_29.stderr parsl.dataflow.dflow INFO: Task 1120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 240c246eecb0c4cc157cd996f9be5970 with result from task 1120 parsl.dataflow.dflow DEBUG: Task 1120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/553d3331-8907-447d-9579-50d6961c257d_isrForCrosstalkSources_1125897_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/553d3331-8907-447d-9579-50d6961c257d_isrForCrosstalkSources_1125897_50.stderr parsl.dataflow.dflow INFO: Task 1019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4275d42931a817dafe46978179e24263 with result from task 1019 parsl.dataflow.dflow DEBUG: Task 1019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/26ce8e7f-b70e-40c8-9b4c-5e4a57f12266_isrForCrosstalkSources_1125928_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/26ce8e7f-b70e-40c8-9b4c-5e4a57f12266_isrForCrosstalkSources_1125928_25.stderr parsl.dataflow.dflow INFO: Task 1113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c34c74d64f41ec58e0a83fb76319b643 with result from task 1113 parsl.dataflow.dflow DEBUG: Task 1113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/6fe3f8df-c3ca-435d-9550-61182a732e64_isrForCrosstalkSources_1125890_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/6fe3f8df-c3ca-435d-9550-61182a732e64_isrForCrosstalkSources_1125890_46.stderr parsl.dataflow.dflow INFO: Task 1020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f05b49ac1115e112ff1d532a71bf685 with result from task 1020 parsl.dataflow.dflow DEBUG: Task 1020 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/0392afca-97a8-4a72-bb3e-dd32823b5521_isrForCrosstalkSources_1125932_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/0392afca-97a8-4a72-bb3e-dd32823b5521_isrForCrosstalkSources_1125932_34.stderr parsl.dataflow.dflow INFO: Task 1123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32be017454d4cf87d6d4d849d6a1bda1 with result from task 1123 parsl.dataflow.dflow DEBUG: Task 1123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/3920175d-b8b2-4d5c-97e0-6c7badff5e94_isrForCrosstalkSources_1125900_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/3920175d-b8b2-4d5c-97e0-6c7badff5e94_isrForCrosstalkSources_1125900_4.stderr parsl.dataflow.dflow INFO: Task 1022 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 936f20d467f0fa810bf08328633be110 with result from task 1022 parsl.dataflow.dflow DEBUG: Task 1022 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/76132605-843e-4f65-8d05-5d0cf323cd70_isrForCrosstalkSources_1125942_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/76132605-843e-4f65-8d05-5d0cf323cd70_isrForCrosstalkSources_1125942_26.stderr parsl.dataflow.dflow INFO: Task 1024 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 546be1f93007b92f32e4accbcf562091 with result from task 1024 parsl.dataflow.dflow DEBUG: Task 1024 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/8b8f944e-e034-49aa-99e9-a82160db5b47_isrForCrosstalkSources_1125942_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/8b8f944e-e034-49aa-99e9-a82160db5b47_isrForCrosstalkSources_1125942_24.stderr parsl.dataflow.dflow INFO: Task 1111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7200f3c2d019c54d7442e0db001f001d with result from task 1111 parsl.dataflow.dflow DEBUG: Task 1111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/45eedce9-eb95-4e9f-aec5-b0e221275458_isrForCrosstalkSources_1125943_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/45eedce9-eb95-4e9f-aec5-b0e221275458_isrForCrosstalkSources_1125943_30.stderr parsl.dataflow.dflow INFO: Task 1124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9094fe8906f3450f82cc66457b4466c9 with result from task 1124 parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:29 proc_lsst.multi:146 [INFO] found job 16896501 in provider checkpoint proc_lsst.multi INFO: found job 16896501 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/2ab85107-1602-40e9-b6ba-419bb72f0230_isrForCrosstalkSources_1125897_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/2ab85107-1602-40e9-b6ba-419bb72f0230_isrForCrosstalkSources_1125897_45.stderr parsl.dataflow.dflow INFO: Task 1109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55546615af048442a502277acbd6c881 with result from task 1109 parsl.dataflow.dflow DEBUG: Task 1109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/29d06600-2000-442a-a992-a32e5e1ba806_isrForCrosstalkSources_1125943_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/29d06600-2000-442a-a992-a32e5e1ba806_isrForCrosstalkSources_1125943_56.stderr parsl.dataflow.dflow INFO: Task 1021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4683ee44f7831f7461f5fc5426422cb9 with result from task 1021 parsl.dataflow.dflow DEBUG: Task 1021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ed700a1f-dab7-4d72-a4d9-6d296281dce7_isrForCrosstalkSources_1125894_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ed700a1f-dab7-4d72-a4d9-6d296281dce7_isrForCrosstalkSources_1125894_36.stderr parsl.dataflow.dflow INFO: Task 1129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a59410ae053250c6e2e6ca4cb75a2536 with result from task 1129 parsl.dataflow.dflow DEBUG: Task 1129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/69378571-32ad-4453-aa02-6363dba5a6d7_isrForCrosstalkSources_1125889_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/69378571-32ad-4453-aa02-6363dba5a6d7_isrForCrosstalkSources_1125889_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:29 proc_lsst.multi:146 [INFO] found job 16896502 in provider checkpoint proc_lsst.multi INFO: found job 16896502 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fe8167b07a858b83bff0f39c653ad67 with result from task 1138 parsl.dataflow.dflow DEBUG: Task 1138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/6d88f5dd-380e-42e3-97a4-4a085b0d1268_isrForCrosstalkSources_1125943_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/6d88f5dd-380e-42e3-97a4-4a085b0d1268_isrForCrosstalkSources_1125943_24.stderr parsl.dataflow.dflow INFO: Task 1127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84afaa4ba087607362f6d818816b11e0 with result from task 1127 parsl.dataflow.dflow DEBUG: Task 1127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e6f3efd6-686b-40c0-a8c0-c77c52c4a34f_isrForCrosstalkSources_1125937_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e6f3efd6-686b-40c0-a8c0-c77c52c4a34f_isrForCrosstalkSources_1125937_40.stderr parsl.dataflow.dflow INFO: Task 1121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82f5fc26dbccc10c0b059dd6976ffc51 with result from task 1121 parsl.dataflow.dflow DEBUG: Task 1121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e7c02a22-8b2c-4d6f-a782-7d668d7e1dcb_isrForCrosstalkSources_1125916_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e7c02a22-8b2c-4d6f-a782-7d668d7e1dcb_isrForCrosstalkSources_1125916_35.stderr parsl.dataflow.dflow INFO: Task 1110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4e214153fbe4f86b2d21436d4916389 with result from task 1110 parsl.dataflow.dflow DEBUG: Task 1110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/574b9e96-8ce9-452d-9de3-10ce93445fe3_isrForCrosstalkSources_1125923_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/574b9e96-8ce9-452d-9de3-10ce93445fe3_isrForCrosstalkSources_1125923_58.stderr parsl.dataflow.dflow INFO: Task 1025 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccf2b41877aeb56f19e4dcb12c828003 with result from task 1025 parsl.dataflow.dflow DEBUG: Task 1025 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ffa58167-38a6-4366-9001-16c975bc2d4d_isrForCrosstalkSources_1125894_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ffa58167-38a6-4366-9001-16c975bc2d4d_isrForCrosstalkSources_1125894_57.stderr parsl.dataflow.dflow INFO: Task 1023 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99464f62c3f079c6613ed566a2ff4314 with result from task 1023 parsl.dataflow.dflow DEBUG: Task 1023 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/0b70f8d2-d877-4a55-abc0-f94e2dcb160c_isrForCrosstalkSources_1125889_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/0b70f8d2-d877-4a55-abc0-f94e2dcb160c_isrForCrosstalkSources_1125889_31.stderr parsl.dataflow.dflow INFO: Task 1126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46698fd2b3974450ef868c258e08feee with result from task 1126 parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:29 proc_lsst.multi:146 [INFO] found job 16896503 in provider checkpoint proc_lsst.multi INFO: found job 16896503 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/d30c4c02-a1f9-4f8a-be00-6a3ddf0bacfd_isrForCrosstalkSources_1125938_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/d30c4c02-a1f9-4f8a-be00-6a3ddf0bacfd_isrForCrosstalkSources_1125938_7.stderr parsl.dataflow.dflow INFO: Task 1139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54a2745807ce58bec78dca1acf93cbe5 with result from task 1139 parsl.dataflow.dflow DEBUG: Task 1139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/bfbbd503-e912-47b4-b3e7-446a739d6bca_isrForCrosstalkSources_1125893_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/bfbbd503-e912-47b4-b3e7-446a739d6bca_isrForCrosstalkSources_1125893_31.stderr parsl.dataflow.dflow INFO: Task 1122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89568ccad34b286850861de06215fa94 with result from task 1122 parsl.dataflow.dflow DEBUG: Task 1122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/f787d9a9-b3b6-479d-a6bf-f526678fac57_isrForCrosstalkSources_1125912_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/f787d9a9-b3b6-479d-a6bf-f526678fac57_isrForCrosstalkSources_1125912_18.stderr parsl.dataflow.dflow INFO: Task 1128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73defe6b79f45e9f06b22202368d6c67 with result from task 1128 parsl.dataflow.dflow DEBUG: Task 1128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8a405fd0-e79d-4b38-996b-175f06c3efa8_isrForCrosstalkSources_1125891_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8a405fd0-e79d-4b38-996b-175f06c3efa8_isrForCrosstalkSources_1125891_1.stderr parsl.dataflow.dflow INFO: Task 1140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fea5c79397f7656875df90a55a93373f with result from task 1140 parsl.dataflow.dflow DEBUG: Task 1140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/f75baec3-7101-4c12-8e83-8c17754e28d1_isrForCrosstalkSources_1125925_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/f75baec3-7101-4c12-8e83-8c17754e28d1_isrForCrosstalkSources_1125925_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:30 proc_lsst.multi:146 [INFO] found job 16896504 in provider checkpoint proc_lsst.multi INFO: found job 16896504 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4d845363459ea318e84c95e07d7cfe2 with result from task 1018 parsl.dataflow.dflow DEBUG: Task 1018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/559e49e8-3016-4cc2-bdb7-aa8d26718285_isrForCrosstalkSources_1125900_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/559e49e8-3016-4cc2-bdb7-aa8d26718285_isrForCrosstalkSources_1125900_61.stderr parsl.dataflow.dflow INFO: Task 1106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4a52ddccaca5ea3ef6bebdd44f25cba with result from task 1106 parsl.dataflow.dflow DEBUG: Task 1106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/76644ed0-586a-4e43-861a-3f1cff5ad722_isrForCrosstalkSources_1125893_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/76644ed0-586a-4e43-861a-3f1cff5ad722_isrForCrosstalkSources_1125893_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:30 proc_lsst.multi:146 [INFO] found job 16896505 in provider checkpoint proc_lsst.multi INFO: found job 16896505 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50221b88eb4f554b976bb8cf8a30bfd6 with result from task 1118 parsl.dataflow.dflow DEBUG: Task 1118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/ee8a0452-9cad-42e2-abee-cfe30571d229_isrForCrosstalkSources_1125938_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/ee8a0452-9cad-42e2-abee-cfe30571d229_isrForCrosstalkSources_1125938_62.stderr parsl.dataflow.dflow INFO: Task 1105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b92403ae3ccc48819262f6c0d5113e0 with result from task 1105 parsl.dataflow.dflow DEBUG: Task 1105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/88f437b2-4e14-40e4-9252-2d3e2f786210_isrForCrosstalkSources_1125921_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/88f437b2-4e14-40e4-9252-2d3e2f786210_isrForCrosstalkSources_1125921_62.stderr parsl.dataflow.dflow INFO: Task 1125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd3692ca93b36c7ffa2ae112229a1415 with result from task 1125 parsl.dataflow.dflow DEBUG: Task 1125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/43ec69ee-3ab4-43f2-a9f8-ccec986069e2_isrForCrosstalkSources_1125894_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/43ec69ee-3ab4-43f2-a9f8-ccec986069e2_isrForCrosstalkSources_1125894_54.stderr parsl.dataflow.dflow INFO: Task 1141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77c9abd323ac8b1303d95abea7079a06 with result from task 1141 parsl.dataflow.dflow DEBUG: Task 1141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/ee7e8d06-559f-42b6-831b-d608392053bb_isrForCrosstalkSources_1125927_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/ee7e8d06-559f-42b6-831b-d608392053bb_isrForCrosstalkSources_1125927_32.stderr parsl.dataflow.dflow INFO: Task 1115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bec2b91845515f70d4bbef53eb811b1a with result from task 1115 parsl.dataflow.dflow DEBUG: Task 1115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/60a08456-5ed6-411f-b5ae-2115f89ae05f_isrForCrosstalkSources_1125942_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/60a08456-5ed6-411f-b5ae-2115f89ae05f_isrForCrosstalkSources_1125942_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:31 proc_lsst.multi:146 [INFO] found job 16896506 in provider checkpoint proc_lsst.multi INFO: found job 16896506 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6806b2c58cf66618f16de6319dc2985 with result from task 1137 parsl.dataflow.dflow DEBUG: Task 1137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/629246da-defe-4cae-91ef-8e847b5dc412_isrForCrosstalkSources_1125894_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/629246da-defe-4cae-91ef-8e847b5dc412_isrForCrosstalkSources_1125894_16.stderr parsl.dataflow.dflow INFO: Task 1133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3cee8193a083aba56810ed71b5057f5 with result from task 1133 parsl.dataflow.dflow DEBUG: Task 1133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/9d34351e-f85d-4308-990c-4ee0749e112b_isrForCrosstalkSources_1125893_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/9d34351e-f85d-4308-990c-4ee0749e112b_isrForCrosstalkSources_1125893_2.stderr parsl.dataflow.dflow INFO: Task 1134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b71ed33efca2e1e62a7ae5015f963821 with result from task 1134 parsl.dataflow.dflow DEBUG: Task 1134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f78f75b6-b7fc-4151-812d-0b1cfbd57ece_isrForCrosstalkSources_1125910_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f78f75b6-b7fc-4151-812d-0b1cfbd57ece_isrForCrosstalkSources_1125910_50.stderr parsl.dataflow.dflow INFO: Task 1132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d304c39e1c46b03bc5face8f5b6ba545 with result from task 1132 parsl.dataflow.dflow DEBUG: Task 1132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/9b166ae8-d90d-4ed6-bccc-727b605bb2bb_isrForCrosstalkSources_1125895_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/9b166ae8-d90d-4ed6-bccc-727b605bb2bb_isrForCrosstalkSources_1125895_12.stderr parsl.dataflow.dflow INFO: Task 1135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d74d19caae2435a580a92145137f3f3d with result from task 1135 parsl.dataflow.dflow DEBUG: Task 1135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/80119163-d0f1-4381-9ca7-f19ee8f64d6b_isrForCrosstalkSources_1125932_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/80119163-d0f1-4381-9ca7-f19ee8f64d6b_isrForCrosstalkSources_1125932_21.stderr parsl.dataflow.dflow INFO: Task 1131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf8919f77a1ed87b1cc69d8034d77c85 with result from task 1131 parsl.dataflow.dflow DEBUG: Task 1131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/e570e47c-1e79-4ed6-9fe9-fb729f5612a8_isrForCrosstalkSources_1125941_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/e570e47c-1e79-4ed6-9fe9-fb729f5612a8_isrForCrosstalkSources_1125941_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:31 proc_lsst.multi:146 [INFO] found job 16896507 in provider checkpoint proc_lsst.multi INFO: found job 16896507 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 316daa1b6d7274d946c03d7d791ff248 with result from task 1143 parsl.dataflow.dflow DEBUG: Task 1143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/dc694308-5dc1-4129-97cb-9e1ac8f9202c_isrForCrosstalkSources_1125921_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/dc694308-5dc1-4129-97cb-9e1ac8f9202c_isrForCrosstalkSources_1125921_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:32 proc_lsst.multi:146 [INFO] found job 16896508 in provider checkpoint proc_lsst.multi INFO: found job 16896508 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e2f57f48325588b0a6798f00d4b037e with result from task 1130 parsl.dataflow.dflow DEBUG: Task 1130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/52470dde-6947-4075-a010-08a7044a85cf_isrForCrosstalkSources_1125917_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/52470dde-6947-4075-a010-08a7044a85cf_isrForCrosstalkSources_1125917_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:32 proc_lsst.multi:146 [INFO] found job 16896509 in provider checkpoint proc_lsst.multi INFO: found job 16896509 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:33 proc_lsst.multi:146 [INFO] found job 16896510 in provider checkpoint proc_lsst.multi INFO: found job 16896510 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:33 proc_lsst.multi:146 [INFO] found job 16896511 in provider checkpoint proc_lsst.multi INFO: found job 16896511 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:33 proc_lsst.multi:146 [INFO] found job 16896519 in provider checkpoint proc_lsst.multi INFO: found job 16896519 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7920fd150b1101f1a739b220ad737aa9 with result from task 1162 parsl.dataflow.dflow DEBUG: Task 1162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/597cc6d5-dc52-4760-8676-e04db801eebd_isrForCrosstalkSources_1125943_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/597cc6d5-dc52-4760-8676-e04db801eebd_isrForCrosstalkSources_1125943_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:34 proc_lsst.multi:146 [INFO] found job 16896520 in provider checkpoint proc_lsst.multi INFO: found job 16896520 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:34 proc_lsst.multi:146 [INFO] found job 16896521 in provider checkpoint proc_lsst.multi INFO: found job 16896521 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ceea56bff037ea53845320e201a476dc with result from task 1142 parsl.dataflow.dflow DEBUG: Task 1142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/376d5a07-6486-44de-bd0a-560d037ea2ab_isrForCrosstalkSources_1125915_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/376d5a07-6486-44de-bd0a-560d037ea2ab_isrForCrosstalkSources_1125915_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:34 proc_lsst.multi:146 [INFO] found job 16896522 in provider checkpoint proc_lsst.multi INFO: found job 16896522 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:34 proc_lsst.multi:146 [INFO] found job 16896523 in provider checkpoint proc_lsst.multi INFO: found job 16896523 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fc25c1595f2ac085ec51de5c6665ff2 with result from task 1164 parsl.dataflow.dflow DEBUG: Task 1164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/8f889799-b4d7-4dda-bb45-a5231f455ae0_isrForCrosstalkSources_1125895_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/8f889799-b4d7-4dda-bb45-a5231f455ae0_isrForCrosstalkSources_1125895_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:35 proc_lsst.multi:146 [INFO] found job 16896524 in provider checkpoint proc_lsst.multi INFO: found job 16896524 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:35 proc_lsst.multi:146 [INFO] found job 16896525 in provider checkpoint proc_lsst.multi INFO: found job 16896525 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:35 proc_lsst.multi:146 [INFO] found job 16896526 in provider checkpoint proc_lsst.multi INFO: found job 16896526 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e79f0fe3362e96d0c3b68d463f9c028 with result from task 1189 parsl.dataflow.dflow DEBUG: Task 1189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/96db6dfc-0593-4c9e-a14c-540116c3e4fe_isrForCrosstalkSources_1125901_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/96db6dfc-0593-4c9e-a14c-540116c3e4fe_isrForCrosstalkSources_1125901_34.stderr parsl.dataflow.dflow INFO: Task 1158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d82a9526d353555411d482be3805f669 with result from task 1158 parsl.dataflow.dflow DEBUG: Task 1158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/caf1fdbe-9372-4fae-a47c-d5e8f0abac23_isrForCrosstalkSources_1125936_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/caf1fdbe-9372-4fae-a47c-d5e8f0abac23_isrForCrosstalkSources_1125936_46.stderr parsl.dataflow.dflow INFO: Task 1150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf4a6019eeaa81e247dc83063ff93f8b with result from task 1150 parsl.dataflow.dflow DEBUG: Task 1150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/95d5b46e-36b9-4a5e-85a9-8046703af6a9_isrForCrosstalkSources_1125927_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/95d5b46e-36b9-4a5e-85a9-8046703af6a9_isrForCrosstalkSources_1125927_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:35 proc_lsst.multi:146 [INFO] found job 16896527 in provider checkpoint proc_lsst.multi INFO: found job 16896527 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ea25964be7b48dbe626abdd381da58a with result from task 1157 parsl.dataflow.dflow DEBUG: Task 1157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/f478f5ed-9047-4d50-8c38-da2b805e3f08_isrForCrosstalkSources_1125944_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/f478f5ed-9047-4d50-8c38-da2b805e3f08_isrForCrosstalkSources_1125944_3.stderr parsl.dataflow.dflow INFO: Task 1147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f45f81bbf2d252a848b73a2e6c8c632 with result from task 1147 parsl.dataflow.dflow DEBUG: Task 1147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/c85e55c5-9363-4c31-ada0-683bb31de7fe_isrForCrosstalkSources_1125901_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/c85e55c5-9363-4c31-ada0-683bb31de7fe_isrForCrosstalkSources_1125901_6.stderr parsl.dataflow.dflow INFO: Task 1152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc4c75e2273dc3e3ccb69ff1865b6c2e with result from task 1152 parsl.dataflow.dflow DEBUG: Task 1152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/7d764871-be75-4cf3-b99a-5ae946c8c206_isrForCrosstalkSources_1125940_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/7d764871-be75-4cf3-b99a-5ae946c8c206_isrForCrosstalkSources_1125940_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:36 proc_lsst.multi:146 [INFO] found job 16896528 in provider checkpoint proc_lsst.multi INFO: found job 16896528 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 891b49feb654de7bd0bda332e42e411d with result from task 1144 parsl.dataflow.dflow DEBUG: Task 1144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/4347d443-6918-48a3-9295-9ed6e9b1c482_isrForCrosstalkSources_1125914_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/4347d443-6918-48a3-9295-9ed6e9b1c482_isrForCrosstalkSources_1125914_12.stderr parsl.dataflow.dflow INFO: Task 1145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7fb5b1e27d4c78d7432bac5cc0cf882 with result from task 1145 parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:36 proc_lsst.multi:146 [INFO] found job 16896529 in provider checkpoint proc_lsst.multi INFO: found job 16896529 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/a7b571e8-38da-4924-bb5b-deaa0ba06065_isrForCrosstalkSources_1125899_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/a7b571e8-38da-4924-bb5b-deaa0ba06065_isrForCrosstalkSources_1125899_10.stderr parsl.dataflow.dflow INFO: Task 1136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2237565c097e3142d1c05a91518309bd with result from task 1136 parsl.dataflow.dflow DEBUG: Task 1136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/e36fe90c-c5f7-4bb5-b0b6-6fea51c1cbb9_isrForCrosstalkSources_1125891_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/e36fe90c-c5f7-4bb5-b0b6-6fea51c1cbb9_isrForCrosstalkSources_1125891_31.stderr parsl.dataflow.dflow INFO: Task 1146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01320f4ba11b42f78e970e1e672ddd9e with result from task 1146 parsl.dataflow.dflow DEBUG: Task 1146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/17a736bf-3aae-4d52-b376-067f7d36df1f_isrForCrosstalkSources_1125908_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/17a736bf-3aae-4d52-b376-067f7d36df1f_isrForCrosstalkSources_1125908_28.stderr parsl.dataflow.dflow INFO: Task 1155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f9c6eef0b57254e4a75fb7b35b4568c with result from task 1155 parsl.dataflow.dflow DEBUG: Task 1155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/d7cb91a9-efe8-4f94-95cd-ebb847100ced_isrForCrosstalkSources_1125947_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/d7cb91a9-efe8-4f94-95cd-ebb847100ced_isrForCrosstalkSources_1125947_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:36 proc_lsst.multi:146 [INFO] found job 16896530 in provider checkpoint proc_lsst.multi INFO: found job 16896530 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27e148236076154a4f1579011c3d735a with result from task 1159 parsl.dataflow.dflow DEBUG: Task 1159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/abb955fb-cddd-445c-9973-26a9f25b46c2_isrForCrosstalkSources_1125890_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/abb955fb-cddd-445c-9973-26a9f25b46c2_isrForCrosstalkSources_1125890_2.stderr parsl.dataflow.dflow INFO: Task 1148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 855ce9c7c249bfe6801738d3a365ed3a with result from task 1148 parsl.dataflow.dflow DEBUG: Task 1148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/e372b0d6-27af-43f6-a559-fd188f7a49d1_isrForCrosstalkSources_1125894_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/e372b0d6-27af-43f6-a559-fd188f7a49d1_isrForCrosstalkSources_1125894_39.stderr parsl.dataflow.dflow INFO: Task 1161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1564754a1d0c8dcdb2f7f179662955f4 with result from task 1161 parsl.dataflow.dflow DEBUG: Task 1161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d09f144f-4f87-4db5-8795-464d7bacdbf5_isrForCrosstalkSources_1125923_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d09f144f-4f87-4db5-8795-464d7bacdbf5_isrForCrosstalkSources_1125923_51.stderr parsl.dataflow.dflow INFO: Task 1160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f405452e109e7cfa78ceb13f9445f389 with result from task 1160 parsl.dataflow.dflow DEBUG: Task 1160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9ed0fe5c-515d-4295-9bec-b4dfd23bf501_isrForCrosstalkSources_1125934_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9ed0fe5c-515d-4295-9bec-b4dfd23bf501_isrForCrosstalkSources_1125934_42.stderr parsl.dataflow.dflow INFO: Task 1163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67fea63724a18a43b90133e77de35af3 with result from task 1163 parsl.dataflow.dflow DEBUG: Task 1163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/2848d32c-3d60-415b-a64f-2883b2579ef1_isrForCrosstalkSources_1125925_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/2848d32c-3d60-415b-a64f-2883b2579ef1_isrForCrosstalkSources_1125925_9.stderr parsl.dataflow.dflow INFO: Task 1156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 242486bfc4b97eaac5314d67411c61a0 with result from task 1156 parsl.dataflow.dflow DEBUG: Task 1156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/af239030-165f-4949-a007-331bdb311f4d_isrForCrosstalkSources_1125932_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/af239030-165f-4949-a007-331bdb311f4d_isrForCrosstalkSources_1125932_45.stderr parsl.dataflow.dflow INFO: Task 1153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37dd9d7afb94e95fe6356a434a2489c4 with result from task 1153 parsl.dataflow.dflow DEBUG: Task 1153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e566e0ee-ac4c-4448-95c7-b3f4c0396747_isrForCrosstalkSources_1125895_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e566e0ee-ac4c-4448-95c7-b3f4c0396747_isrForCrosstalkSources_1125895_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:37 proc_lsst.multi:146 [INFO] found job 16896531 in provider checkpoint proc_lsst.multi INFO: found job 16896531 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 861494feda3bcafff64001573c898fd3 with result from task 1151 parsl.dataflow.dflow DEBUG: Task 1151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/3a6cfc8d-5706-4144-93de-4e7ed7e83f90_isrForCrosstalkSources_1125947_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/3a6cfc8d-5706-4144-93de-4e7ed7e83f90_isrForCrosstalkSources_1125947_5.stderr parsl.dataflow.dflow INFO: Task 1149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43e2443579fb830a6c1ff0d86c1dbbb9 with result from task 1149 parsl.dataflow.dflow DEBUG: Task 1149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/fb75aa13-e179-4e90-8cc4-6b0cc8be54fd_isrForCrosstalkSources_1125935_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/fb75aa13-e179-4e90-8cc4-6b0cc8be54fd_isrForCrosstalkSources_1125935_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:37 proc_lsst.multi:146 [INFO] found job 16896533 in provider checkpoint proc_lsst.multi INFO: found job 16896533 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:37 proc_lsst.multi:146 [INFO] found job 16896534 in provider checkpoint proc_lsst.multi INFO: found job 16896534 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bb79be3a58e6be036affee54692ee95 with result from task 1154 parsl.dataflow.dflow DEBUG: Task 1154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/a1e157f5-b397-4d40-aa4b-5c0ac4f89f99_isrForCrosstalkSources_1125939_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/a1e157f5-b397-4d40-aa4b-5c0ac4f89f99_isrForCrosstalkSources_1125939_48.stderr parsl.dataflow.dflow INFO: Task 1239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe59ee3b9804d121181656bb771ff00d with result from task 1239 parsl.dataflow.dflow DEBUG: Task 1239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/c97a2607-bb5a-435e-b07a-4a31a8e3af48_isrForCrosstalkSources_1125924_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/c97a2607-bb5a-435e-b07a-4a31a8e3af48_isrForCrosstalkSources_1125924_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:38 proc_lsst.multi:146 [INFO] found job 16896535 in provider checkpoint proc_lsst.multi INFO: found job 16896535 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2033fb822561bfed406515e42c117e79 with result from task 1214 parsl.dataflow.dflow DEBUG: Task 1214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/5ff2d7d5-cf66-4689-8db3-d44f6f6455d5_isrForCrosstalkSources_1125900_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/5ff2d7d5-cf66-4689-8db3-d44f6f6455d5_isrForCrosstalkSources_1125900_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:38 proc_lsst.multi:146 [INFO] found job 16896536 in provider checkpoint proc_lsst.multi INFO: found job 16896536 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44f31f15c1b5330ce0482dfaa68e4dcc with result from task 1294 parsl.dataflow.dflow DEBUG: Task 1294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/87ecce0c-27f2-408c-bcb6-f16042da2671_isrForCrosstalkSources_1125903_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/87ecce0c-27f2-408c-bcb6-f16042da2671_isrForCrosstalkSources_1125903_33.stderr parsl.dataflow.dflow INFO: Task 1367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e9e6611ea27a5258576fae0ad1424cd with result from task 1367 parsl.dataflow.dflow DEBUG: Task 1367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/eabec1bc-566e-45c6-9af7-a474bbe49d24_isrForCrosstalkSources_1125919_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/eabec1bc-566e-45c6-9af7-a474bbe49d24_isrForCrosstalkSources_1125919_27.stderr parsl.dataflow.dflow INFO: Task 1368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffc142dcb608997c8d9f95f98710cc67 with result from task 1368 parsl.dataflow.dflow DEBUG: Task 1368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/65c8c05e-bc34-4a81-9ecf-ef6f44d6f76e_isrForCrosstalkSources_1125912_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/65c8c05e-bc34-4a81-9ecf-ef6f44d6f76e_isrForCrosstalkSources_1125912_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:39 proc_lsst.multi:146 [INFO] found job 16896537 in provider checkpoint proc_lsst.multi INFO: found job 16896537 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0399fcb56c2944f6155e0e57b40d1104 with result from task 1358 parsl.dataflow.dflow DEBUG: Task 1358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/3e1bb95d-908a-4f48-a68b-5ce0a4e53137_isrForCrosstalkSources_1125901_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/3e1bb95d-908a-4f48-a68b-5ce0a4e53137_isrForCrosstalkSources_1125901_8.stderr parsl.dataflow.dflow INFO: Task 1311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfad54a8a2d4ed1efd15072801b93c08 with result from task 1311 parsl.dataflow.dflow DEBUG: Task 1311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/ac729294-fa15-4b4b-888b-ae1b08487095_isrForCrosstalkSources_1125931_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/ac729294-fa15-4b4b-888b-ae1b08487095_isrForCrosstalkSources_1125931_4.stderr parsl.dataflow.dflow INFO: Task 1281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae24937b6fc30e2183622bb3328d313a with result from task 1281 parsl.dataflow.dflow DEBUG: Task 1281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/96a5d255-bb66-46be-b6c6-5d5aa295d4c4_isrForCrosstalkSources_1125900_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/96a5d255-bb66-46be-b6c6-5d5aa295d4c4_isrForCrosstalkSources_1125900_30.stderr parsl.dataflow.dflow INFO: Task 1347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 277e4335768addc1a7868390726ef6f1 with result from task 1347 parsl.dataflow.dflow DEBUG: Task 1347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/58be2540-7f15-4b20-8e6f-d65eeb9ed255_isrForCrosstalkSources_1125940_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/58be2540-7f15-4b20-8e6f-d65eeb9ed255_isrForCrosstalkSources_1125940_40.stderr parsl.dataflow.dflow INFO: Task 1286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7702d4e50b3856642dc483e0fcd38b83 with result from task 1286 parsl.dataflow.dflow DEBUG: Task 1286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/d06956f4-38ee-4b18-9701-607a187bf1f0_isrForCrosstalkSources_1125894_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/d06956f4-38ee-4b18-9701-607a187bf1f0_isrForCrosstalkSources_1125894_17.stderr parsl.dataflow.dflow INFO: Task 1385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d82c253615fcd45c43fcc8c001a2acd with result from task 1385 parsl.dataflow.dflow DEBUG: Task 1385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e8aa9e70-814b-4737-9f32-8ba08f9831bd_isrForCrosstalkSources_1125937_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e8aa9e70-814b-4737-9f32-8ba08f9831bd_isrForCrosstalkSources_1125937_9.stderr parsl.dataflow.dflow INFO: Task 1348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e0c5e095badd090a469583fee356b68 with result from task 1348 parsl.dataflow.dflow DEBUG: Task 1348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/5c49a2cf-3a7e-48cf-af6c-b8ae7c7424df_isrForCrosstalkSources_1125927_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/5c49a2cf-3a7e-48cf-af6c-b8ae7c7424df_isrForCrosstalkSources_1125927_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 1290 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry c702599e6777d96446dbfe258a0a289a with result from task 1290 parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 1290 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/7fff316d-d714-4de8-82bf-a738c7cadb87_isrForCrosstalkSources_1125944_56.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/7fff316d-d714-4de8-82bf-a738c7cadb87_isrForCrosstalkSources_1125944_56.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:39 proc_lsst.multi:146 [INFO] found job 16896538 in provider checkpoint proc_lsst.multi INFO: found job 16896538 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16d11c760feb8b1c65ff0641e4d5a73c with result from task 1289 parsl.dataflow.dflow DEBUG: Task 1289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/739fcc1e-a0ba-440c-803f-71926b290a6b_isrForCrosstalkSources_1125932_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/739fcc1e-a0ba-440c-803f-71926b290a6b_isrForCrosstalkSources_1125932_35.stderr parsl.dataflow.dflow INFO: Task 1387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04d9b1f270d53895524c4a2be6da693c with result from task 1387 parsl.dataflow.dflow DEBUG: Task 1387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/bd0a4bb1-6e25-479f-9d66-dab1c40e8e8a_isrForCrosstalkSources_1125915_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/bd0a4bb1-6e25-479f-9d66-dab1c40e8e8a_isrForCrosstalkSources_1125915_20.stderr parsl.dataflow.dflow INFO: Task 1285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0dd5f2863b2c99a9579e25645a676f12 with result from task 1285 parsl.dataflow.dflow DEBUG: Task 1285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/2a3360ca-bf6a-4902-aaa9-1c17177ca2c7_isrForCrosstalkSources_1125941_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/2a3360ca-bf6a-4902-aaa9-1c17177ca2c7_isrForCrosstalkSources_1125941_62.stderr parsl.dataflow.dflow INFO: Task 1283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31abb85b85c9de620729ff76c0c425fb with result from task 1283 parsl.dataflow.dflow DEBUG: Task 1283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/47d78eaf-9621-437e-bcc4-a8bb97478edd_isrForCrosstalkSources_1125915_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/47d78eaf-9621-437e-bcc4-a8bb97478edd_isrForCrosstalkSources_1125915_36.stderr parsl.dataflow.dflow INFO: Task 1384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2af815750afe13eb246551b514cb328 with result from task 1384 parsl.dataflow.dflow DEBUG: Task 1384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/5e8d9d7f-41a7-4f50-b9c8-fe1e07375114_isrForCrosstalkSources_1125905_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/5e8d9d7f-41a7-4f50-b9c8-fe1e07375114_isrForCrosstalkSources_1125905_51.stderr parsl.dataflow.dflow INFO: Task 1292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3108cc951734a484dacb0c45f1613db with result from task 1292 parsl.dataflow.dflow DEBUG: Task 1292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/50bc19b7-ad86-47a8-a0fb-6f3ff2038d9c_isrForCrosstalkSources_1125919_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/50bc19b7-ad86-47a8-a0fb-6f3ff2038d9c_isrForCrosstalkSources_1125919_17.stderr parsl.dataflow.dflow INFO: Task 1388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0772f88234ffd27339446b380057b1c8 with result from task 1388 parsl.dataflow.dflow DEBUG: Task 1388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/adb1e325-f23b-45f5-a863-3c6352ec9be0_isrForCrosstalkSources_1125914_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/adb1e325-f23b-45f5-a863-3c6352ec9be0_isrForCrosstalkSources_1125914_57.stderr parsl.dataflow.dflow INFO: Task 1369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fcd80f1fbe62c3080c3247ddebad944 with result from task 1369 parsl.dataflow.dflow DEBUG: Task 1369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d73ec0d6-0925-4a60-b130-75f7494cf85a_isrForCrosstalkSources_1125890_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d73ec0d6-0925-4a60-b130-75f7494cf85a_isrForCrosstalkSources_1125890_16.stderr parsl.dataflow.dflow INFO: Task 1287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68e7f3a5ae94b6588a8c49747969f17d with result from task 1287 parsl.dataflow.dflow DEBUG: Task 1287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/5f519bc6-1370-40ed-bac9-3728eff84364_isrForCrosstalkSources_1125917_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/5f519bc6-1370-40ed-bac9-3728eff84364_isrForCrosstalkSources_1125917_2.stderr parsl.dataflow.dflow INFO: Task 1280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e2620209a12d96e6eab57d5e9c69e85 with result from task 1280 parsl.dataflow.dflow DEBUG: Task 1280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/c9c58b0f-8bfc-4a00-824d-027fa62a3aa5_isrForCrosstalkSources_1125939_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/c9c58b0f-8bfc-4a00-824d-027fa62a3aa5_isrForCrosstalkSources_1125939_61.stderr parsl.dataflow.dflow INFO: Task 1370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba35095a9e15e440f051b35bb2ac5461 with result from task 1370 parsl.dataflow.dflow DEBUG: Task 1370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/dac9dc43-dcce-49b2-9c12-1213156e7dc6_isrForCrosstalkSources_1125913_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/dac9dc43-dcce-49b2-9c12-1213156e7dc6_isrForCrosstalkSources_1125913_13.stderr parsl.dataflow.dflow INFO: Task 1282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 094c79543268fb9815de9923890f6f90 with result from task 1282 parsl.dataflow.dflow DEBUG: Task 1282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/060946aa-457a-4f50-baa3-00f0469ff05b_isrForCrosstalkSources_1125901_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/060946aa-457a-4f50-baa3-00f0469ff05b_isrForCrosstalkSources_1125901_49.stderr parsl.dataflow.dflow INFO: Task 1383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e13d170adcebfd01f639dda8015a3cac with result from task 1383 parsl.dataflow.dflow DEBUG: Task 1383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/667a6410-f3fe-4cff-ab0c-92372e874397_isrForCrosstalkSources_1125911_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/667a6410-f3fe-4cff-ab0c-92372e874397_isrForCrosstalkSources_1125911_51.stderr parsl.dataflow.dflow INFO: Task 1320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de8a33b7248292dc44a1ade19d8f78f8 with result from task 1320 parsl.dataflow.dflow DEBUG: Task 1320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/8bd6d832-b78a-44ca-bcd8-45dfc71684c3_isrForCrosstalkSources_1125924_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/8bd6d832-b78a-44ca-bcd8-45dfc71684c3_isrForCrosstalkSources_1125924_50.stderr parsl.dataflow.dflow INFO: Task 1373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 441450609ce89c3f1a674fced532bb7f with result from task 1373 parsl.dataflow.dflow DEBUG: Task 1373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/342a14f6-c88a-4d60-be71-66c6c7b18088_isrForCrosstalkSources_1125899_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/342a14f6-c88a-4d60-be71-66c6c7b18088_isrForCrosstalkSources_1125899_1.stderr parsl.dataflow.dflow INFO: Task 1293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5b14a3b326ead06ef2a97f52d3b2bfc with result from task 1293 parsl.dataflow.dflow DEBUG: Task 1293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6187af46-11da-41cc-b0dc-77158b1b3c79_isrForCrosstalkSources_1125927_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6187af46-11da-41cc-b0dc-77158b1b3c79_isrForCrosstalkSources_1125927_44.stderr parsl.dataflow.dflow INFO: Task 1321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75188a0bffb87aeb4e0cf5faaaea39eb with result from task 1321 parsl.dataflow.dflow DEBUG: Task 1321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/b695cf2e-0add-4d04-bcbd-f932af6ae2e9_isrForCrosstalkSources_1125940_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/b695cf2e-0add-4d04-bcbd-f932af6ae2e9_isrForCrosstalkSources_1125940_12.stderr parsl.dataflow.dflow INFO: Task 1357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3410c9fcc96c13b8b5b32b82789f240 with result from task 1357 parsl.dataflow.dflow DEBUG: Task 1357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/5f3e1adc-353b-411f-aa30-7b7e2060e275_isrForCrosstalkSources_1125940_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/5f3e1adc-353b-411f-aa30-7b7e2060e275_isrForCrosstalkSources_1125940_49.stderr parsl.dataflow.dflow INFO: Task 1382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dc395325ad0f9397bb846d3047f9373 with result from task 1382 parsl.dataflow.dflow DEBUG: Task 1382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/9ac60577-87cf-43c3-a8fb-ad098dc88d64_isrForCrosstalkSources_1125917_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/9ac60577-87cf-43c3-a8fb-ad098dc88d64_isrForCrosstalkSources_1125917_38.stderr parsl.dataflow.dflow INFO: Task 1399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 343021e59f5defa98629aed08d9560a5 with result from task 1399 parsl.dataflow.dflow DEBUG: Task 1399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/1b884800-95ea-4c4f-ad1d-5586c02b0953_isrForCrosstalkSources_1125920_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/1b884800-95ea-4c4f-ad1d-5586c02b0953_isrForCrosstalkSources_1125920_48.stderr parsl.dataflow.dflow INFO: Task 1397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 575e9f7869c66475855467ce175b2276 with result from task 1397 parsl.dataflow.dflow DEBUG: Task 1397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/e4b8cb45-2295-469d-82ef-bc8bae9cb6ae_isrForCrosstalkSources_1125920_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/e4b8cb45-2295-469d-82ef-bc8bae9cb6ae_isrForCrosstalkSources_1125920_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:40 proc_lsst.multi:146 [INFO] found job 16896539 in provider checkpoint proc_lsst.multi INFO: found job 16896539 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 775f9e41f55977327eb45a6cf1b130a5 with result from task 1291 parsl.dataflow.dflow DEBUG: Task 1291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a03a9713-7b3c-4d97-b32d-e45e82c49e10_isrForCrosstalkSources_1125937_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a03a9713-7b3c-4d97-b32d-e45e82c49e10_isrForCrosstalkSources_1125937_35.stderr parsl.dataflow.dflow INFO: Task 1284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c64b720ab7cb502cde74979c35a9feb0 with result from task 1284 parsl.dataflow.dflow DEBUG: Task 1284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/efd5e822-54f5-427d-904b-929451dd16bc_isrForCrosstalkSources_1125943_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/efd5e822-54f5-427d-904b-929451dd16bc_isrForCrosstalkSources_1125943_28.stderr parsl.dataflow.dflow INFO: Task 1428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff2946bb2152dd0d97d12550787df70f with result from task 1428 parsl.dataflow.dflow DEBUG: Task 1428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/3398de58-64f8-4f15-8d92-025a78a21357_isrForCrosstalkSources_1125917_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/3398de58-64f8-4f15-8d92-025a78a21357_isrForCrosstalkSources_1125917_24.stderr parsl.dataflow.dflow INFO: Task 1372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c9c8d71f1ac9e804d62c353e5f71bb9 with result from task 1372 parsl.dataflow.dflow DEBUG: Task 1372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/2ff8ffb7-0ab3-41fe-9504-ab3ffe235b99_isrForCrosstalkSources_1125937_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/2ff8ffb7-0ab3-41fe-9504-ab3ffe235b99_isrForCrosstalkSources_1125937_24.stderr parsl.dataflow.dflow INFO: Task 1409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 894ce4c93bb7180b75d6f13e839994b2 with result from task 1409 parsl.dataflow.dflow DEBUG: Task 1409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/fa57b67e-c264-41e7-b7fd-160984be1c75_isrForCrosstalkSources_1125923_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/fa57b67e-c264-41e7-b7fd-160984be1c75_isrForCrosstalkSources_1125923_8.stderr parsl.dataflow.dflow INFO: Task 1371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a85603243d7e18d7e3eb09cc83bdeaf1 with result from task 1371 parsl.dataflow.dflow DEBUG: Task 1371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/a686a204-7a05-4614-b870-55034ea05ac8_isrForCrosstalkSources_1125891_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/a686a204-7a05-4614-b870-55034ea05ac8_isrForCrosstalkSources_1125891_41.stderr parsl.dataflow.dflow INFO: Task 1418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c01a7be38dd1b587bb4c9a1f15177abb with result from task 1418 parsl.dataflow.dflow DEBUG: Task 1418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/4eaa5b11-89ae-4bb9-bb66-ed820b54ae21_isrForCrosstalkSources_1125899_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/4eaa5b11-89ae-4bb9-bb66-ed820b54ae21_isrForCrosstalkSources_1125899_62.stderr parsl.dataflow.dflow INFO: Task 1288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d8449503ff5318c18bae622daa72540 with result from task 1288 parsl.dataflow.dflow DEBUG: Task 1288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/e7815104-22d4-41f0-9ae6-64900868bb2d_isrForCrosstalkSources_1125906_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/e7815104-22d4-41f0-9ae6-64900868bb2d_isrForCrosstalkSources_1125906_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:41 proc_lsst.multi:146 [INFO] found job 16896540 in provider checkpoint proc_lsst.multi INFO: found job 16896540 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f22caec0266f82c6fa7c88c21a857dcb with result from task 1322 parsl.dataflow.dflow DEBUG: Task 1322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/6eba5f12-1272-48d2-9096-3cc79e2ca48e_isrForCrosstalkSources_1125914_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/6eba5f12-1272-48d2-9096-3cc79e2ca48e_isrForCrosstalkSources_1125914_61.stderr parsl.dataflow.dflow INFO: Task 1429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 389a38376d84658d9f48e593eaafca60 with result from task 1429 parsl.dataflow.dflow DEBUG: Task 1429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/f12fb531-d5cd-4926-9bd7-fae37d6542a0_isrForCrosstalkSources_1125936_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/f12fb531-d5cd-4926-9bd7-fae37d6542a0_isrForCrosstalkSources_1125936_23.stderr parsl.dataflow.dflow INFO: Task 1398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 058b7b6a0979f040104a0f7d7003ff40 with result from task 1398 parsl.dataflow.dflow DEBUG: Task 1398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/e8702108-2978-4a1d-9206-d5664c5dbab0_isrForCrosstalkSources_1125896_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/e8702108-2978-4a1d-9206-d5664c5dbab0_isrForCrosstalkSources_1125896_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:41 proc_lsst.multi:146 [INFO] found job 16896541 in provider checkpoint proc_lsst.multi INFO: found job 16896541 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9887b7a06b11b18ba707b24c8f43e014 with result from task 1427 parsl.dataflow.dflow DEBUG: Task 1427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/4a38cf8c-b920-4dce-9971-899838765d89_isrForCrosstalkSources_1125909_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/4a38cf8c-b920-4dce-9971-899838765d89_isrForCrosstalkSources_1125909_36.stderr parsl.dataflow.dflow INFO: Task 1430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08e4fa7913de32f75168d05d091c47d6 with result from task 1430 parsl.dataflow.dflow DEBUG: Task 1430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/eb27a66b-8b0f-42c8-9037-434024351594_isrForCrosstalkSources_1125945_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/eb27a66b-8b0f-42c8-9037-434024351594_isrForCrosstalkSources_1125945_43.stderr parsl.dataflow.dflow INFO: Task 1408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62c0f01d18419428428d8c73d27ff0a4 with result from task 1408 parsl.dataflow.dflow DEBUG: Task 1408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/7966bd7e-ee34-46a4-9577-0dcbd70eb6a6_isrForCrosstalkSources_1125894_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/7966bd7e-ee34-46a4-9577-0dcbd70eb6a6_isrForCrosstalkSources_1125894_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:42 proc_lsst.multi:146 [INFO] found job 16896542 in provider checkpoint proc_lsst.multi INFO: found job 16896542 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 147d0cd92352b43092522f6e3b67d4a4 with result from task 1431 parsl.dataflow.dflow DEBUG: Task 1431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/b66689b4-a84e-40e3-a667-a1e6548304ca_isrForCrosstalkSources_1125925_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/b66689b4-a84e-40e3-a667-a1e6548304ca_isrForCrosstalkSources_1125925_61.stderr parsl.dataflow.dflow INFO: Task 1386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c2e62eb117924e96d75a467074602fd with result from task 1386 parsl.dataflow.dflow DEBUG: Task 1386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/384eb45d-7d50-419f-a499-1fc4c1ecdeab_isrForCrosstalkSources_1125904_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/384eb45d-7d50-419f-a499-1fc4c1ecdeab_isrForCrosstalkSources_1125904_58.stderr parsl.dataflow.dflow INFO: Task 1433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27244f0de6155df8e3460bbecb2f8926 with result from task 1433 parsl.dataflow.dflow DEBUG: Task 1433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e052108a-79f3-45d7-a217-b1a19519bd21_isrForCrosstalkSources_1125901_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e052108a-79f3-45d7-a217-b1a19519bd21_isrForCrosstalkSources_1125901_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:43 proc_lsst.multi:146 [INFO] found job 16896543 in provider checkpoint proc_lsst.multi INFO: found job 16896543 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e4a0a2126fe5c9628dd61f99b02a35b with result from task 1435 parsl.dataflow.dflow DEBUG: Task 1435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f9e159e2-89c4-4eb4-9021-7b9517c6d635_isrForCrosstalkSources_1125894_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f9e159e2-89c4-4eb4-9021-7b9517c6d635_isrForCrosstalkSources_1125894_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:43 proc_lsst.multi:146 [INFO] found job 16896544 in provider checkpoint proc_lsst.multi INFO: found job 16896544 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdffc8bd2f413375258e16cddc292780 with result from task 1436 parsl.dataflow.dflow DEBUG: Task 1436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/8149d404-e66a-4489-8a4d-db7d58bbe7ad_isrForCrosstalkSources_1125914_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/8149d404-e66a-4489-8a4d-db7d58bbe7ad_isrForCrosstalkSources_1125914_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:44 proc_lsst.multi:146 [INFO] found job 16896545 in provider checkpoint proc_lsst.multi INFO: found job 16896545 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:44 proc_lsst.multi:146 [INFO] found job 16896546 in provider checkpoint proc_lsst.multi INFO: found job 16896546 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e4e425dd6739824e5a21f73c5003f42 with result from task 1432 parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:44 proc_lsst.multi:146 [INFO] found job 16896547 in provider checkpoint proc_lsst.multi INFO: found job 16896547 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/42da528b-54f7-4f15-890d-741252b67356_isrForCrosstalkSources_1125926_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/42da528b-54f7-4f15-890d-741252b67356_isrForCrosstalkSources_1125926_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:45 proc_lsst.multi:146 [INFO] found job 16896548 in provider checkpoint proc_lsst.multi INFO: found job 16896548 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:46 proc_lsst.multi:146 [INFO] found job 16896549 in provider checkpoint proc_lsst.multi INFO: found job 16896549 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:46 proc_lsst.multi:146 [INFO] found job 16896550 in provider checkpoint proc_lsst.multi INFO: found job 16896550 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 060312b283627d651074f5bb301b8975 with result from task 1450 parsl.dataflow.dflow DEBUG: Task 1450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/d40ece24-1f68-45ec-bb30-2bb300575c97_isrForCrosstalkSources_1125899_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/d40ece24-1f68-45ec-bb30-2bb300575c97_isrForCrosstalkSources_1125899_27.stderr parsl.dataflow.dflow INFO: Task 1455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1680d4814ea202a1e02c518ead87aa33 with result from task 1455 parsl.dataflow.dflow DEBUG: Task 1455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/ee0f86ed-977c-49b8-a219-03bcd3dc1e47_isrForCrosstalkSources_1125942_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/ee0f86ed-977c-49b8-a219-03bcd3dc1e47_isrForCrosstalkSources_1125942_25.stderr parsl.dataflow.dflow INFO: Task 1438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4bb36cd34e46bda793f0f2e955c04a9 with result from task 1438 parsl.dataflow.dflow DEBUG: Task 1438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/b94a94d2-88f5-4ec4-892f-2a01bcf355c9_isrForCrosstalkSources_1125898_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/b94a94d2-88f5-4ec4-892f-2a01bcf355c9_isrForCrosstalkSources_1125898_15.stderr parsl.dataflow.dflow INFO: Task 1452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ef5fc5b9bd4cfec99d58777d523371c with result from task 1452 parsl.dataflow.dflow DEBUG: Task 1452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/9148f562-5964-47d1-aa9a-71fbcf68cd86_isrForCrosstalkSources_1125914_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/9148f562-5964-47d1-aa9a-71fbcf68cd86_isrForCrosstalkSources_1125914_22.stderr parsl.dataflow.dflow INFO: Task 1454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3e4a4b79801c4e92c7bb1fb3be92142 with result from task 1454 parsl.dataflow.dflow DEBUG: Task 1454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/95b7b277-2b8e-4dfd-97d1-d74e7b5e7c2f_isrForCrosstalkSources_1125896_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/95b7b277-2b8e-4dfd-97d1-d74e7b5e7c2f_isrForCrosstalkSources_1125896_16.stderr parsl.dataflow.dflow INFO: Task 1434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ded80ce767420affebb6128a06af83ba with result from task 1434 parsl.dataflow.dflow DEBUG: Task 1434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/f7c878c0-e0f5-4766-864b-4475401670a5_isrForCrosstalkSources_1125940_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/f7c878c0-e0f5-4766-864b-4475401670a5_isrForCrosstalkSources_1125940_57.stderr parsl.dataflow.dflow INFO: Task 1451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ada201e6f904e819b7cb3aabd3d37fb with result from task 1451 parsl.dataflow.dflow DEBUG: Task 1451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/d2980e08-e0d9-4ce1-aedf-801c77b604f3_isrForCrosstalkSources_1125906_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/d2980e08-e0d9-4ce1-aedf-801c77b604f3_isrForCrosstalkSources_1125906_5.stderr parsl.dataflow.dflow INFO: Task 1456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4baec6aa547b143fe3646a9361481bb with result from task 1456 parsl.dataflow.dflow DEBUG: Task 1456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/3855454e-39b9-4daa-b694-061f338a3ee7_isrForCrosstalkSources_1125932_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/3855454e-39b9-4daa-b694-061f338a3ee7_isrForCrosstalkSources_1125932_6.stderr parsl.dataflow.dflow INFO: Task 1447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f276da838b4d6d38ccde9106ef94229f with result from task 1447 parsl.dataflow.dflow DEBUG: Task 1447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/df9b8254-6462-4ab3-9ec6-8086e6f2965d_isrForCrosstalkSources_1125907_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/df9b8254-6462-4ab3-9ec6-8086e6f2965d_isrForCrosstalkSources_1125907_10.stderr parsl.dataflow.dflow INFO: Task 1441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f293e8e21b556d2decf8e620d1e1c9b3 with result from task 1441 parsl.dataflow.dflow DEBUG: Task 1441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/416e83cd-4c20-40cf-8766-5dcd82724596_isrForCrosstalkSources_1125899_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/416e83cd-4c20-40cf-8766-5dcd82724596_isrForCrosstalkSources_1125899_41.stderr parsl.dataflow.dflow INFO: Task 1437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ba5e0414bcdb0eae80f0560219045b8 with result from task 1437 parsl.dataflow.dflow DEBUG: Task 1437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/7ededb62-216b-48fa-8a71-edc07919c426_isrForCrosstalkSources_1125940_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/7ededb62-216b-48fa-8a71-edc07919c426_isrForCrosstalkSources_1125940_24.stderr parsl.dataflow.dflow INFO: Task 1439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69b0ef1e22b47af3f33bcf9aa40163da with result from task 1439 parsl.dataflow.dflow DEBUG: Task 1439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/1783cf9d-4682-4e28-9595-738fe6d2bad5_isrForCrosstalkSources_1125926_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/1783cf9d-4682-4e28-9595-738fe6d2bad5_isrForCrosstalkSources_1125926_35.stderr parsl.dataflow.dflow INFO: Task 1448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90f7782b5d7cec0524161e028345d1dd with result from task 1448 parsl.dataflow.dflow DEBUG: Task 1448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/ff3a7339-b938-4fc4-af97-7fdd44eecdcd_isrForCrosstalkSources_1125942_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/ff3a7339-b938-4fc4-af97-7fdd44eecdcd_isrForCrosstalkSources_1125942_7.stderr parsl.dataflow.dflow INFO: Task 1449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4b195d6535f38283618e6a9a97f99e0 with result from task 1449 parsl.dataflow.dflow DEBUG: Task 1449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e3411b2d-730e-49bd-bf17-9b82c7253e0d_isrForCrosstalkSources_1125929_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e3411b2d-730e-49bd-bf17-9b82c7253e0d_isrForCrosstalkSources_1125929_61.stderr parsl.dataflow.dflow INFO: Task 1461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29a7562b7cb0c875d0fdf45036294519 with result from task 1461 parsl.dataflow.dflow DEBUG: Task 1461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/4943839c-be8f-4f66-9d74-800131a89117_isrForCrosstalkSources_1125943_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/4943839c-be8f-4f66-9d74-800131a89117_isrForCrosstalkSources_1125943_26.stderr parsl.dataflow.dflow INFO: Task 1453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d6ff5662c324b477a3c7b6e8238196a with result from task 1453 parsl.dataflow.dflow DEBUG: Task 1453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/aa01237a-f63a-406a-adc6-7219224648bf_isrForCrosstalkSources_1125907_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/aa01237a-f63a-406a-adc6-7219224648bf_isrForCrosstalkSources_1125907_47.stderr parsl.dataflow.dflow INFO: Task 1457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd2eec2d27357d4b0921139f194687ef with result from task 1457 parsl.dataflow.dflow DEBUG: Task 1457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/d5097aeb-f4d2-4dcb-807a-4fb4f09ffdbe_isrForCrosstalkSources_1125898_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/d5097aeb-f4d2-4dcb-807a-4fb4f09ffdbe_isrForCrosstalkSources_1125898_49.stderr parsl.dataflow.dflow INFO: Task 1460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ecae06b5f5143070bff5a30d33b2ce8 with result from task 1460 parsl.dataflow.dflow DEBUG: Task 1460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/81c051ed-3aec-49d4-9cf9-f44aec9f5d95_isrForCrosstalkSources_1125934_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/81c051ed-3aec-49d4-9cf9-f44aec9f5d95_isrForCrosstalkSources_1125934_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:48 proc_lsst.multi:146 [INFO] found job 16896551 in provider checkpoint proc_lsst.multi INFO: found job 16896551 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 540d2b0f13188066e6a99eac19e5d239 with result from task 1445 parsl.dataflow.dflow DEBUG: Task 1445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/6e72c7e1-8fa3-447b-991f-0f64dc2f2831_isrForCrosstalkSources_1125946_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/6e72c7e1-8fa3-447b-991f-0f64dc2f2831_isrForCrosstalkSources_1125946_43.stderr parsl.dataflow.dflow INFO: Task 1443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25a38e58baccd0d3b06a2c75d008021e with result from task 1443 parsl.dataflow.dflow DEBUG: Task 1443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/a6a20078-eab6-40e3-845f-30be93ed24dd_isrForCrosstalkSources_1125912_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/a6a20078-eab6-40e3-845f-30be93ed24dd_isrForCrosstalkSources_1125912_49.stderr parsl.dataflow.dflow INFO: Task 1467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7583fab85571534a322127a24c0e2df7 with result from task 1467 parsl.dataflow.dflow DEBUG: Task 1467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/8568bf08-4335-46db-9e6a-b27a73277567_isrForCrosstalkSources_1125946_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/8568bf08-4335-46db-9e6a-b27a73277567_isrForCrosstalkSources_1125946_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:49 proc_lsst.multi:146 [INFO] found job 16896552 in provider checkpoint proc_lsst.multi INFO: found job 16896552 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df94fa37e228a86668f67f7018788690 with result from task 1444 parsl.dataflow.dflow DEBUG: Task 1444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d8585084-9214-4526-94ba-fb915ba6a2bc_isrForCrosstalkSources_1125934_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d8585084-9214-4526-94ba-fb915ba6a2bc_isrForCrosstalkSources_1125934_59.stderr parsl.dataflow.dflow INFO: Task 1462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6522b72eed7793108bd8cc82bdf155c4 with result from task 1462 parsl.dataflow.dflow DEBUG: Task 1462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/2749c00c-c1be-426a-9927-7825a2ef7901_isrForCrosstalkSources_1125904_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/2749c00c-c1be-426a-9927-7825a2ef7901_isrForCrosstalkSources_1125904_53.stderr parsl.dataflow.dflow INFO: Task 1440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d63465228997bf65f445ac61a45dde9 with result from task 1440 parsl.dataflow.dflow DEBUG: Task 1440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b1776f1e-a9fa-4dcc-bef8-e31d36053d8e_isrForCrosstalkSources_1125938_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b1776f1e-a9fa-4dcc-bef8-e31d36053d8e_isrForCrosstalkSources_1125938_49.stderr parsl.dataflow.dflow INFO: Task 1446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b28f2c4931e844e6cab027885d5ab9a with result from task 1446 parsl.dataflow.dflow DEBUG: Task 1446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/b2e9fb23-1a05-43a2-b4f6-a752537a4ee9_isrForCrosstalkSources_1125928_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/b2e9fb23-1a05-43a2-b4f6-a752537a4ee9_isrForCrosstalkSources_1125928_30.stderr parsl.dataflow.dflow INFO: Task 1458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 778b32a314f3668c3637079e75901e87 with result from task 1458 parsl.dataflow.dflow DEBUG: Task 1458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/8a86f2d4-f69d-4f82-9ad1-1c28893f699d_isrForCrosstalkSources_1125942_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/8a86f2d4-f69d-4f82-9ad1-1c28893f699d_isrForCrosstalkSources_1125942_13.stderr parsl.dataflow.dflow INFO: Task 1474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79a9d924fc3a5b9f76f695bec6b74bbd with result from task 1474 parsl.dataflow.dflow DEBUG: Task 1474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/09687c14-90a7-41d4-adf0-ef5e99df3dd0_isrForCrosstalkSources_1125935_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/09687c14-90a7-41d4-adf0-ef5e99df3dd0_isrForCrosstalkSources_1125935_18.stderr parsl.dataflow.dflow INFO: Task 1442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9aee546e994a18e0efcb5f134db5148 with result from task 1442 parsl.dataflow.dflow DEBUG: Task 1442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/ed946c67-9d54-46eb-8d96-5065d88b548c_isrForCrosstalkSources_1125916_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/ed946c67-9d54-46eb-8d96-5065d88b548c_isrForCrosstalkSources_1125916_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:49 proc_lsst.multi:146 [INFO] found job 16896553 in provider checkpoint proc_lsst.multi INFO: found job 16896553 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f241f63bf2c95df920948ab8163dd2a with result from task 1463 parsl.dataflow.dflow DEBUG: Task 1463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/c0179c14-e195-4cc8-a450-2f3d5f0445e9_isrForCrosstalkSources_1125925_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/c0179c14-e195-4cc8-a450-2f3d5f0445e9_isrForCrosstalkSources_1125925_40.stderr parsl.dataflow.dflow INFO: Task 1490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80b9b284d7b16ed40e9be682a88c9c84 with result from task 1490 parsl.dataflow.dflow DEBUG: Task 1490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/53ecf6bb-ec88-443a-a126-e2aa9c9a2aa7_isrForCrosstalkSources_1125945_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/53ecf6bb-ec88-443a-a126-e2aa9c9a2aa7_isrForCrosstalkSources_1125945_12.stderr parsl.dataflow.dflow INFO: Task 1471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c76da174b7ba651d7e4963420c7dadb9 with result from task 1471 parsl.dataflow.dflow DEBUG: Task 1471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/6a06123d-c633-4f11-8ce6-a407d8c613ca_isrForCrosstalkSources_1125897_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/6a06123d-c633-4f11-8ce6-a407d8c613ca_isrForCrosstalkSources_1125897_14.stderr parsl.dataflow.dflow INFO: Task 1482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa3e258fb4ed973f721a56fe4be16ca4 with result from task 1482 parsl.dataflow.dflow DEBUG: Task 1482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/af3ab0da-f791-4bc1-95ba-9f56c6fbce41_isrForCrosstalkSources_1125904_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/af3ab0da-f791-4bc1-95ba-9f56c6fbce41_isrForCrosstalkSources_1125904_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:50 proc_lsst.multi:146 [INFO] found job 16896554 in provider checkpoint proc_lsst.multi INFO: found job 16896554 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4e690323fb41df3c463712fa5c1e4af with result from task 1489 parsl.dataflow.dflow DEBUG: Task 1489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/c086f50c-878f-44d6-9c00-1111ae65e5ac_isrForCrosstalkSources_1125913_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/c086f50c-878f-44d6-9c00-1111ae65e5ac_isrForCrosstalkSources_1125913_1.stderr parsl.dataflow.dflow INFO: Task 1488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aca5cbe8631efd05197f49f015270827 with result from task 1488 parsl.dataflow.dflow DEBUG: Task 1488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/d5374de7-7d19-4ee1-a2e1-2f8ed32d754c_isrForCrosstalkSources_1125945_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/d5374de7-7d19-4ee1-a2e1-2f8ed32d754c_isrForCrosstalkSources_1125945_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:50 proc_lsst.multi:146 [INFO] found job 16896555 in provider checkpoint proc_lsst.multi INFO: found job 16896555 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42ca893a4336bd05be16fdc3e5c11430 with result from task 1468 parsl.dataflow.dflow DEBUG: Task 1468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a00f8c53-1e92-4964-bf13-4518db91611b_isrForCrosstalkSources_1125937_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a00f8c53-1e92-4964-bf13-4518db91611b_isrForCrosstalkSources_1125937_18.stderr parsl.dataflow.dflow INFO: Task 1485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63f67723f927b7ade2ca17cca1413956 with result from task 1485 parsl.dataflow.dflow DEBUG: Task 1485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/593d914a-ff96-40c4-8e0d-9e549de26648_isrForCrosstalkSources_1125889_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/593d914a-ff96-40c4-8e0d-9e549de26648_isrForCrosstalkSources_1125889_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:50 proc_lsst.multi:146 [INFO] found job 16896556 in provider checkpoint proc_lsst.multi INFO: found job 16896556 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd993a31784e12722cc2bd23b6564e65 with result from task 1495 parsl.dataflow.dflow DEBUG: Task 1495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/9477656f-8822-456d-9c20-d1412166a10a_isrForCrosstalkSources_1125910_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/9477656f-8822-456d-9c20-d1412166a10a_isrForCrosstalkSources_1125910_30.stderr parsl.dataflow.dflow INFO: Task 1501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4d839ff9ec51f641fedf19e26f94742 with result from task 1501 parsl.dataflow.dflow DEBUG: Task 1501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/3e8f0078-c745-43ee-bdb7-5b2a7c36ad22_isrForCrosstalkSources_1125904_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/3e8f0078-c745-43ee-bdb7-5b2a7c36ad22_isrForCrosstalkSources_1125904_36.stderr parsl.dataflow.dflow INFO: Task 1487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c77559cd266176e38c423a4880eba79 with result from task 1487 parsl.dataflow.dflow DEBUG: Task 1487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/7d654db1-6cb5-49be-bd75-3272f6679e73_isrForCrosstalkSources_1125911_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/7d654db1-6cb5-49be-bd75-3272f6679e73_isrForCrosstalkSources_1125911_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:51 proc_lsst.multi:146 [INFO] found job 16896559 in provider checkpoint proc_lsst.multi INFO: found job 16896559 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 522f0cd0c190ff93113dd182b95fc2e5 with result from task 1491 parsl.dataflow.dflow DEBUG: Task 1491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/57164fa4-77cd-42d2-a5bc-0239c58915eb_isrForCrosstalkSources_1125914_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/57164fa4-77cd-42d2-a5bc-0239c58915eb_isrForCrosstalkSources_1125914_2.stderr parsl.dataflow.dflow INFO: Task 1476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec5b8a871b113ea2873c35ae103f86e1 with result from task 1476 parsl.dataflow.dflow DEBUG: Task 1476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/e72e8fd5-a942-4c0b-bb42-dc63eab26be2_isrForCrosstalkSources_1125890_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/e72e8fd5-a942-4c0b-bb42-dc63eab26be2_isrForCrosstalkSources_1125890_52.stderr parsl.dataflow.dflow INFO: Task 1493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 279ffa3ec368087ea3c742b7a4e3df85 with result from task 1493 parsl.dataflow.dflow DEBUG: Task 1493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/8d629632-2176-4e35-8b4a-7bc72940fac8_isrForCrosstalkSources_1125946_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/8d629632-2176-4e35-8b4a-7bc72940fac8_isrForCrosstalkSources_1125946_30.stderr parsl.dataflow.dflow INFO: Task 1470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b15ccb311ebfb1577fb51bb1acaac1c2 with result from task 1470 parsl.dataflow.dflow DEBUG: Task 1470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/fe5f9fb0-a1cb-4283-9a48-f1ac00e6180c_isrForCrosstalkSources_1125931_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/fe5f9fb0-a1cb-4283-9a48-f1ac00e6180c_isrForCrosstalkSources_1125931_5.stderr parsl.dataflow.dflow INFO: Task 1498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56850711d724bac817a5af2faf4edc05 with result from task 1498 parsl.dataflow.dflow DEBUG: Task 1498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/bcf922f9-5fca-4b93-b7df-18dc44c20fd7_isrForCrosstalkSources_1125942_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/bcf922f9-5fca-4b93-b7df-18dc44c20fd7_isrForCrosstalkSources_1125942_20.stderr parsl.dataflow.dflow INFO: Task 1502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd3aa03bebd771cea423e91c9ed5b92c with result from task 1502 parsl.dataflow.dflow DEBUG: Task 1502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/2518428f-06d0-4851-9a5c-e00826e7aa13_isrForCrosstalkSources_1125906_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/2518428f-06d0-4851-9a5c-e00826e7aa13_isrForCrosstalkSources_1125906_51.stderr parsl.dataflow.dflow INFO: Task 1497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47830ecc0023b9cbe42b8472d12dba58 with result from task 1497 parsl.dataflow.dflow DEBUG: Task 1497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/cd01063d-8538-49a6-948f-4a6d9ebaff38_isrForCrosstalkSources_1125919_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/cd01063d-8538-49a6-948f-4a6d9ebaff38_isrForCrosstalkSources_1125919_10.stderr parsl.dataflow.dflow INFO: Task 1480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36e3e2f1e6daeaa5a41850c0e8652c54 with result from task 1480 parsl.dataflow.dflow DEBUG: Task 1480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/674a7ce4-d172-4da7-97a6-b663b7825f20_isrForCrosstalkSources_1125924_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/674a7ce4-d172-4da7-97a6-b663b7825f20_isrForCrosstalkSources_1125924_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:51 proc_lsst.multi:146 [INFO] found job 16896561 in provider checkpoint proc_lsst.multi INFO: found job 16896561 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f9a0f3c5e0825ed63d50b518851a55f with result from task 1500 parsl.dataflow.dflow DEBUG: Task 1500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f66faf50-5c6c-4762-9ec9-551178d17155_isrForCrosstalkSources_1125922_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f66faf50-5c6c-4762-9ec9-551178d17155_isrForCrosstalkSources_1125922_11.stderr parsl.dataflow.dflow INFO: Task 1477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3462554fc26458d3918faaf4ce300a86 with result from task 1477 parsl.dataflow.dflow DEBUG: Task 1477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/d5ac29bb-f7ab-4062-a979-c1c6ede28931_isrForCrosstalkSources_1125910_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/d5ac29bb-f7ab-4062-a979-c1c6ede28931_isrForCrosstalkSources_1125910_49.stderr parsl.dataflow.dflow INFO: Task 1483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d3f46f019017fc76c439622b3567ad9 with result from task 1483 parsl.dataflow.dflow DEBUG: Task 1483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/5fcbdd50-d586-4967-b4da-32df4bcde12d_isrForCrosstalkSources_1125892_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/5fcbdd50-d586-4967-b4da-32df4bcde12d_isrForCrosstalkSources_1125892_51.stderr parsl.dataflow.dflow INFO: Task 1478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18167a8aa40bfe827f1734d244cb9a95 with result from task 1478 parsl.dataflow.dflow DEBUG: Task 1478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/a9b47bdc-fd48-4a29-9797-6868acc3fb05_isrForCrosstalkSources_1125918_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/a9b47bdc-fd48-4a29-9797-6868acc3fb05_isrForCrosstalkSources_1125918_46.stderr parsl.dataflow.dflow INFO: Task 1465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b27720ae4e03bf49961590fffb094a35 with result from task 1465 parsl.dataflow.dflow DEBUG: Task 1465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/2bb730a9-0798-48f7-be97-6c28d8a5e49a_isrForCrosstalkSources_1125947_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/2bb730a9-0798-48f7-be97-6c28d8a5e49a_isrForCrosstalkSources_1125947_58.stderr parsl.dataflow.dflow INFO: Task 1469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a53e5737b4694f27a122fb5c42af7778 with result from task 1469 parsl.dataflow.dflow DEBUG: Task 1469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/c003e0cf-918c-4ff4-9ecf-65956f068c95_isrForCrosstalkSources_1125908_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/c003e0cf-918c-4ff4-9ecf-65956f068c95_isrForCrosstalkSources_1125908_58.stderr parsl.dataflow.dflow INFO: Task 1494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9724f90fa58bafd7d47b8a75861bb216 with result from task 1494 parsl.dataflow.dflow DEBUG: Task 1494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/cfa5dd94-592a-4c39-b1bd-58fb1ca82a97_isrForCrosstalkSources_1125935_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/cfa5dd94-592a-4c39-b1bd-58fb1ca82a97_isrForCrosstalkSources_1125935_26.stderr parsl.dataflow.dflow INFO: Task 1479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd73744f142186c158a8a648bb685c02 with result from task 1479 parsl.dataflow.dflow DEBUG: Task 1479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/fd3b4dcf-48c8-4b14-884d-666fcf181399_isrForCrosstalkSources_1125933_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/fd3b4dcf-48c8-4b14-884d-666fcf181399_isrForCrosstalkSources_1125933_17.stderr parsl.dataflow.dflow INFO: Task 1503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6b7977096f1174c677e82b15fd76310 with result from task 1503 parsl.dataflow.dflow DEBUG: Task 1503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/dc71e1ee-011f-4eb2-87d2-27f1f7d596dc_isrForCrosstalkSources_1125915_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/dc71e1ee-011f-4eb2-87d2-27f1f7d596dc_isrForCrosstalkSources_1125915_1.stderr parsl.dataflow.dflow INFO: Task 1486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e06707e4bfdce8a33df4cedd91f5860b with result from task 1486 parsl.dataflow.dflow DEBUG: Task 1486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/d59693e0-eabe-469e-975e-8c29f125575b_isrForCrosstalkSources_1125944_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/d59693e0-eabe-469e-975e-8c29f125575b_isrForCrosstalkSources_1125944_41.stderr parsl.dataflow.dflow INFO: Task 1459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d169f1ce270f071d4febc9e4050d7f55 with result from task 1459 parsl.dataflow.dflow DEBUG: Task 1459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/85dd2bd9-ec9a-4268-8b18-d3f4c624f8f0_isrForCrosstalkSources_1125898_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/85dd2bd9-ec9a-4268-8b18-d3f4c624f8f0_isrForCrosstalkSources_1125898_37.stderr parsl.dataflow.dflow INFO: Task 1472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d44a3f36065f0cba416f3cf89e9c66a with result from task 1472 parsl.dataflow.dflow DEBUG: Task 1472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/3a17492a-6eca-4830-bdd5-4f2b5b8f6ae2_isrForCrosstalkSources_1125934_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/3a17492a-6eca-4830-bdd5-4f2b5b8f6ae2_isrForCrosstalkSources_1125934_56.stderr parsl.dataflow.dflow INFO: Task 1475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f67a76cb6e03bbd8cc6bee223db5d4c with result from task 1475 parsl.dataflow.dflow DEBUG: Task 1475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/1d184128-e9da-4a26-94b2-496d95ec1b3e_isrForCrosstalkSources_1125898_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/1d184128-e9da-4a26-94b2-496d95ec1b3e_isrForCrosstalkSources_1125898_3.stderr parsl.dataflow.dflow INFO: Task 1466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6024cf70fdf3ba41f7f5b739828f0273 with result from task 1466 parsl.dataflow.dflow DEBUG: Task 1466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/7d3fb1ba-9b4f-4e42-8066-30411a15da31_isrForCrosstalkSources_1125937_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/7d3fb1ba-9b4f-4e42-8066-30411a15da31_isrForCrosstalkSources_1125937_10.stderr parsl.dataflow.dflow INFO: Task 1481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3236a14d12b6f277d4630e5b38770225 with result from task 1481 parsl.dataflow.dflow DEBUG: Task 1481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/dd3d3710-0776-4b07-bd9e-ea4dde547408_isrForCrosstalkSources_1125916_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/dd3d3710-0776-4b07-bd9e-ea4dde547408_isrForCrosstalkSources_1125916_2.stderr parsl.dataflow.dflow INFO: Task 1505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 259488bc564e71cb45b822f613507e24 with result from task 1505 parsl.dataflow.dflow DEBUG: Task 1505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/4edea987-a558-4704-96ba-60eecbd0747a_isrForCrosstalkSources_1125928_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/4edea987-a558-4704-96ba-60eecbd0747a_isrForCrosstalkSources_1125928_15.stderr parsl.dataflow.dflow INFO: Task 1464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03f3d46fc8b82fa3d53be7f05f9e5712 with result from task 1464 parsl.dataflow.dflow DEBUG: Task 1464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/2febfe70-8bb1-4a94-a312-ba0513a737b1_isrForCrosstalkSources_1125916_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/2febfe70-8bb1-4a94-a312-ba0513a737b1_isrForCrosstalkSources_1125916_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:52 proc_lsst.multi:146 [INFO] found job 16896562 in provider checkpoint proc_lsst.multi INFO: found job 16896562 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0222deebf61b25e83682d219baa871dc with result from task 1484 parsl.dataflow.dflow DEBUG: Task 1484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/a98db295-010d-4b99-9894-769905a96f8c_isrForCrosstalkSources_1125944_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/a98db295-010d-4b99-9894-769905a96f8c_isrForCrosstalkSources_1125944_57.stderr parsl.dataflow.dflow INFO: Task 1492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d717386388be33db28bc6832bac1dba2 with result from task 1492 parsl.dataflow.dflow DEBUG: Task 1492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/4e839d56-16fa-4bd7-bfb1-f662114d8dc5_isrForCrosstalkSources_1125895_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/4e839d56-16fa-4bd7-bfb1-f662114d8dc5_isrForCrosstalkSources_1125895_51.stderr parsl.dataflow.dflow INFO: Task 1508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 052f2602ccc05c25cfe75e9a8d7cff8f with result from task 1508 parsl.dataflow.dflow DEBUG: Task 1508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/dcab9d89-f0dc-4f9d-aaf8-0f2f7663cf93_isrForCrosstalkSources_1125920_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/dcab9d89-f0dc-4f9d-aaf8-0f2f7663cf93_isrForCrosstalkSources_1125920_24.stderr parsl.dataflow.dflow INFO: Task 1509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 242f5549148cf82ea10e2d56c590e8fc with result from task 1509 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:52 proc_lsst.multi:146 [INFO] found job 16896563 in provider checkpoint proc_lsst.multi INFO: found job 16896563 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/4d39bac8-0981-4337-8102-5a903a2692ea_isrForCrosstalkSources_1125904_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/4d39bac8-0981-4337-8102-5a903a2692ea_isrForCrosstalkSources_1125904_14.stderr parsl.dataflow.dflow INFO: Task 1507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5e7742071d74edbd02ff96a1650d4c5 with result from task 1507 parsl.dataflow.dflow DEBUG: Task 1507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d03eb2b3-d4be-4d6e-a7db-3bcc5ab9c10e_isrForCrosstalkSources_1125905_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d03eb2b3-d4be-4d6e-a7db-3bcc5ab9c10e_isrForCrosstalkSources_1125905_38.stderr parsl.dataflow.dflow INFO: Task 1473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1bd44340f06dfa94a2e01d525928b14 with result from task 1473 parsl.dataflow.dflow DEBUG: Task 1473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d5dc0c21-dc12-4b71-913f-95ecf5cb30b5_isrForCrosstalkSources_1125911_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d5dc0c21-dc12-4b71-913f-95ecf5cb30b5_isrForCrosstalkSources_1125911_58.stderr parsl.dataflow.dflow INFO: Task 1499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec6d38445986d3b7cf1be3ae37f23982 with result from task 1499 parsl.dataflow.dflow DEBUG: Task 1499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/095f79e7-ff75-4b3e-b934-8c6e6aba92ae_isrForCrosstalkSources_1125918_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/095f79e7-ff75-4b3e-b934-8c6e6aba92ae_isrForCrosstalkSources_1125918_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:53 proc_lsst.multi:146 [INFO] found job 16896564 in provider checkpoint proc_lsst.multi INFO: found job 16896564 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 329f237aa97ea3ae43772c588d39bfb3 with result from task 1506 parsl.dataflow.dflow DEBUG: Task 1506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/8d54148b-0c6d-42d2-85bd-81ef459b55fe_isrForCrosstalkSources_1125904_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/8d54148b-0c6d-42d2-85bd-81ef459b55fe_isrForCrosstalkSources_1125904_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:53 proc_lsst.multi:146 [INFO] found job 16896565 in provider checkpoint proc_lsst.multi INFO: found job 16896565 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1255989858ce87b7214f54c68ef5ea7 with result from task 1496 parsl.dataflow.dflow DEBUG: Task 1496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/41ed0c1d-7c44-456c-9165-a42a8262c128_isrForCrosstalkSources_1125896_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/41ed0c1d-7c44-456c-9165-a42a8262c128_isrForCrosstalkSources_1125896_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:54 proc_lsst.multi:146 [INFO] found job 16896566 in provider checkpoint proc_lsst.multi INFO: found job 16896566 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:55 proc_lsst.multi:146 [INFO] found job 16896568 in provider checkpoint proc_lsst.multi INFO: found job 16896568 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c522e30515d144f7ca00572423ef171 with result from task 1504 parsl.dataflow.dflow DEBUG: Task 1504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/27b4e139-6db0-4082-a005-ced4e2078892_isrForCrosstalkSources_1125947_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/27b4e139-6db0-4082-a005-ced4e2078892_isrForCrosstalkSources_1125947_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:55 proc_lsst.multi:146 [INFO] found job 16896569 in provider checkpoint proc_lsst.multi INFO: found job 16896569 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:56 proc_lsst.multi:146 [INFO] found job 16896570 in provider checkpoint proc_lsst.multi INFO: found job 16896570 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:56 proc_lsst.multi:146 [INFO] found job 16896571 in provider checkpoint proc_lsst.multi INFO: found job 16896571 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:56 proc_lsst.multi:146 [INFO] found job 16896572 in provider checkpoint proc_lsst.multi INFO: found job 16896572 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:56 proc_lsst.multi:146 [INFO] found job 16896573 in provider checkpoint proc_lsst.multi INFO: found job 16896573 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:56 proc_lsst.multi:146 [INFO] found job 16896574 in provider checkpoint proc_lsst.multi INFO: found job 16896574 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:57 proc_lsst.multi:146 [INFO] found job 16896575 in provider checkpoint proc_lsst.multi INFO: found job 16896575 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:57 proc_lsst.multi:146 [INFO] found job 16896576 in provider checkpoint proc_lsst.multi INFO: found job 16896576 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 851bfffab539467ed023f294ac6b89fb with result from task 1510 parsl.dataflow.dflow DEBUG: Task 1510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/3cf337d3-7a3d-47f6-a718-7de7081dccf3_isrForCrosstalkSources_1125945_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/3cf337d3-7a3d-47f6-a718-7de7081dccf3_isrForCrosstalkSources_1125945_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:57 proc_lsst.multi:146 [INFO] found job 16896577 in provider checkpoint proc_lsst.multi INFO: found job 16896577 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:57 proc_lsst.multi:146 [INFO] found job 16896578 in provider checkpoint proc_lsst.multi INFO: found job 16896578 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:58 proc_lsst.multi:146 [INFO] found job 16896579 in provider checkpoint proc_lsst.multi INFO: found job 16896579 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:58 proc_lsst.multi:146 [INFO] found job 16896580 in provider checkpoint proc_lsst.multi INFO: found job 16896580 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:58 proc_lsst.multi:146 [INFO] found job 16896581 in provider checkpoint proc_lsst.multi INFO: found job 16896581 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce9e494f9af8046a6b9c7964541d6b5e with result from task 1537 parsl.dataflow.dflow DEBUG: Task 1537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/6a313fab-05d2-4d76-a55a-0d2f75838129_isrForCrosstalkSources_1125946_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/6a313fab-05d2-4d76-a55a-0d2f75838129_isrForCrosstalkSources_1125946_12.stderr parsl.dataflow.dflow INFO: Task 1525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb03d98cf2d556311272f3486511a5a0 with result from task 1525 parsl.dataflow.dflow DEBUG: Task 1525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/697fb45f-bcef-483f-9bdf-1d2cad352ac4_isrForCrosstalkSources_1125925_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/697fb45f-bcef-483f-9bdf-1d2cad352ac4_isrForCrosstalkSources_1125925_33.stderr parsl.dataflow.dflow INFO: Task 1528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dabe79eb756d25fbadecb4c4e40ff136 with result from task 1528 parsl.dataflow.dflow DEBUG: Task 1528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7ebeba42-00e8-434a-acc0-17eceb52526b_isrForCrosstalkSources_1125929_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7ebeba42-00e8-434a-acc0-17eceb52526b_isrForCrosstalkSources_1125929_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:58 proc_lsst.multi:146 [INFO] found job 16896582 in provider checkpoint proc_lsst.multi INFO: found job 16896582 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:58 proc_lsst.multi:146 [INFO] found job 16896583 in provider checkpoint proc_lsst.multi INFO: found job 16896583 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4154049c2efa68df4f56bdc0b2a08897 with result from task 1206 parsl.dataflow.dflow DEBUG: Task 1206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/7982fa75-57c0-4d70-86d1-3c3a639b1e93_isrForCrosstalkSources_1125910_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/7982fa75-57c0-4d70-86d1-3c3a639b1e93_isrForCrosstalkSources_1125910_5.stderr parsl.dataflow.dflow INFO: Task 1209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e015e2d69c9a543d57dd6baa0f272b9a with result from task 1209 parsl.dataflow.dflow DEBUG: Task 1209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/049160fd-eca5-42cd-b289-b89e542253cc_isrForCrosstalkSources_1125926_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/049160fd-eca5-42cd-b289-b89e542253cc_isrForCrosstalkSources_1125926_46.stderr parsl.dataflow.dflow INFO: Task 1212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f8727559eb1a81741e429469734749f with result from task 1212 parsl.dataflow.dflow DEBUG: Task 1212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0b8bc14d-6be8-4d98-964c-774743515d75_isrForCrosstalkSources_1125914_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0b8bc14d-6be8-4d98-964c-774743515d75_isrForCrosstalkSources_1125914_14.stderr parsl.dataflow.dflow INFO: Task 1207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38fa3683684cd481e7ca3edc6c2f1d1d with result from task 1207 parsl.dataflow.dflow DEBUG: Task 1207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/e2476425-1cfa-4b9e-9034-f925a5c26d07_isrForCrosstalkSources_1125924_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/e2476425-1cfa-4b9e-9034-f925a5c26d07_isrForCrosstalkSources_1125924_23.stderr parsl.dataflow.dflow INFO: Task 1208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3827029ac7c36f43637ee6deab475b38 with result from task 1208 parsl.dataflow.dflow DEBUG: Task 1208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/28aeaad6-0c71-4a08-aadb-9277eee579ea_isrForCrosstalkSources_1125933_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/28aeaad6-0c71-4a08-aadb-9277eee579ea_isrForCrosstalkSources_1125933_38.stderr parsl.dataflow.dflow INFO: Task 1210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a877454493ae52a49243219bda1c9175 with result from task 1210 parsl.dataflow.dflow DEBUG: Task 1210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/83a03753-f3a8-47d6-bc61-0065edadcb63_isrForCrosstalkSources_1125944_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/83a03753-f3a8-47d6-bc61-0065edadcb63_isrForCrosstalkSources_1125944_44.stderr parsl.dataflow.dflow INFO: Task 1213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7256c9c5ee58b7c78cee9dfa5b818eb6 with result from task 1213 parsl.dataflow.dflow DEBUG: Task 1213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/d70273ee-ccb1-4d6f-a53a-632f9747fe87_isrForCrosstalkSources_1125935_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/d70273ee-ccb1-4d6f-a53a-632f9747fe87_isrForCrosstalkSources_1125935_31.stderr parsl.dataflow.dflow INFO: Task 1211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be531ec42e668b4d790e3df6e31d2515 with result from task 1211 parsl.dataflow.dflow DEBUG: Task 1211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/ab1a3914-3abe-489c-92cd-25894f27052d_isrForCrosstalkSources_1125930_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/ab1a3914-3abe-489c-92cd-25894f27052d_isrForCrosstalkSources_1125930_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:59 proc_lsst.multi:146 [INFO] found job 16896584 in provider checkpoint proc_lsst.multi INFO: found job 16896584 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:59 proc_lsst.multi:146 [INFO] found job 16896585 in provider checkpoint proc_lsst.multi INFO: found job 16896585 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:12:59 proc_lsst.multi:146 [INFO] found job 16896586 in provider checkpoint proc_lsst.multi INFO: found job 16896586 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c6a11e3e70a6728d9670ae61dbb571b with result from task 1530 parsl.dataflow.dflow DEBUG: Task 1530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/cbe1d291-50fa-47af-8738-05fc06a96f37_isrForCrosstalkSources_1125919_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/cbe1d291-50fa-47af-8738-05fc06a96f37_isrForCrosstalkSources_1125919_34.stderr parsl.dataflow.dflow INFO: Task 1533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f20e0b093c244d28aecb97c86577b7a with result from task 1533 parsl.dataflow.dflow DEBUG: Task 1533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/cd703c02-faa7-4c93-a389-09e49a06881f_isrForCrosstalkSources_1125944_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/cd703c02-faa7-4c93-a389-09e49a06881f_isrForCrosstalkSources_1125944_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:00 proc_lsst.multi:146 [INFO] found job 16896587 in provider checkpoint proc_lsst.multi INFO: found job 16896587 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75bd69b36e20645c2574da72af190ac2 with result from task 1512 parsl.dataflow.dflow DEBUG: Task 1512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/4a047409-d56a-4803-b615-63745db0ac8c_isrForCrosstalkSources_1125896_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/4a047409-d56a-4803-b615-63745db0ac8c_isrForCrosstalkSources_1125896_45.stderr parsl.dataflow.dflow INFO: Task 1516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08714656d99f64fee3b858bb537fd214 with result from task 1516 parsl.dataflow.dflow DEBUG: Task 1516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/30e06081-d6fe-4e31-9424-95d60fdc3b2a_isrForCrosstalkSources_1125897_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/30e06081-d6fe-4e31-9424-95d60fdc3b2a_isrForCrosstalkSources_1125897_42.stderr parsl.dataflow.dflow INFO: Task 1538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a789998351a6e74dc83cf208931050b with result from task 1538 parsl.dataflow.dflow DEBUG: Task 1538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/ef81d17e-c45c-4777-a6b0-d82118b39102_isrForCrosstalkSources_1125898_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/ef81d17e-c45c-4777-a6b0-d82118b39102_isrForCrosstalkSources_1125898_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:00 proc_lsst.multi:146 [INFO] found job 16896588 in provider checkpoint proc_lsst.multi INFO: found job 16896588 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41e6d0c8ed2086dda4320b64d999f60a with result from task 1558 parsl.dataflow.dflow DEBUG: Task 1558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/464e6e95-b412-468d-9723-95068817b8a0_isrForCrosstalkSources_1125911_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/464e6e95-b412-468d-9723-95068817b8a0_isrForCrosstalkSources_1125911_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:00 proc_lsst.multi:146 [INFO] found job 16896589 in provider checkpoint proc_lsst.multi INFO: found job 16896589 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4874b0f552f13ef4773dbe8c85aab74 with result from task 1531 parsl.dataflow.dflow DEBUG: Task 1531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/c1aed873-6776-4ed3-bfd6-e8dbeacb7619_isrForCrosstalkSources_1125890_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/c1aed873-6776-4ed3-bfd6-e8dbeacb7619_isrForCrosstalkSources_1125890_42.stderr parsl.dataflow.dflow INFO: Task 1543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9d787b4a1c699c11834e606782101df with result from task 1543 parsl.dataflow.dflow DEBUG: Task 1543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/23faffcd-6270-4b43-bc2f-6bd258eff2de_isrForCrosstalkSources_1125936_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/23faffcd-6270-4b43-bc2f-6bd258eff2de_isrForCrosstalkSources_1125936_28.stderr parsl.dataflow.dflow INFO: Task 1266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74395487302bba76866b55d4db30f589 with result from task 1266 parsl.dataflow.dflow DEBUG: Task 1266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/1a6aae29-e0ea-4255-83a4-00fd3942d7f6_isrForCrosstalkSources_1125903_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/1a6aae29-e0ea-4255-83a4-00fd3942d7f6_isrForCrosstalkSources_1125903_29.stderr parsl.dataflow.dflow INFO: Task 1271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c0b1b63f94ab3f8fea17d79d5c87138 with result from task 1271 parsl.dataflow.dflow DEBUG: Task 1271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e6f2b599-4888-4623-bfc6-aa2fd27b9968_isrForCrosstalkSources_1125942_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e6f2b599-4888-4623-bfc6-aa2fd27b9968_isrForCrosstalkSources_1125942_16.stderr parsl.dataflow.dflow INFO: Task 1265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8aeb181b45e78bfc52d5d1bd9afca4f with result from task 1265 parsl.dataflow.dflow DEBUG: Task 1265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c1f2de04-24dc-4dff-8dad-42bfca12ba7c_isrForCrosstalkSources_1125904_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c1f2de04-24dc-4dff-8dad-42bfca12ba7c_isrForCrosstalkSources_1125904_5.stderr parsl.dataflow.dflow INFO: Task 1264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3da97d2029ff7af1026856d3514ef72a with result from task 1264 parsl.dataflow.dflow DEBUG: Task 1264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/c4861fa5-4cfd-4021-8162-f9123050e969_isrForCrosstalkSources_1125931_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/c4861fa5-4cfd-4021-8162-f9123050e969_isrForCrosstalkSources_1125931_7.stderr parsl.dataflow.dflow INFO: Task 1270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0034b796d8c67faa3c8837ea1589b8e1 with result from task 1270 parsl.dataflow.dflow DEBUG: Task 1270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/16c183e1-0a16-4065-9917-ee0ff7ab4789_isrForCrosstalkSources_1125899_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/16c183e1-0a16-4065-9917-ee0ff7ab4789_isrForCrosstalkSources_1125899_29.stderr parsl.dataflow.dflow INFO: Task 1268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be0caf07cae1c6061704855ffcb24826 with result from task 1268 parsl.dataflow.dflow DEBUG: Task 1268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/c31f6900-87dc-433b-8c93-8230a3fbb0aa_isrForCrosstalkSources_1125921_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/c31f6900-87dc-433b-8c93-8230a3fbb0aa_isrForCrosstalkSources_1125921_47.stderr parsl.dataflow.dflow INFO: Task 1269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25a77b6d8777ab2573b54325a013dce9 with result from task 1269 parsl.dataflow.dflow DEBUG: Task 1269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/21e9cf96-a605-4281-a843-c05f3f8d16a3_isrForCrosstalkSources_1125937_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/21e9cf96-a605-4281-a843-c05f3f8d16a3_isrForCrosstalkSources_1125937_12.stderr parsl.dataflow.dflow INFO: Task 1267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 358ec3cbf547d163e0b1f29e72c2d169 with result from task 1267 parsl.dataflow.dflow DEBUG: Task 1267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/03fb2389-9231-4e0d-88c8-4ffb8067da3f_isrForCrosstalkSources_1125938_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/03fb2389-9231-4e0d-88c8-4ffb8067da3f_isrForCrosstalkSources_1125938_50.stderr parsl.dataflow.dflow INFO: Task 1274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 860c94584abb55e9493aadb19021ec66 with result from task 1274 parsl.dataflow.dflow DEBUG: Task 1274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/413baf6e-cfe2-41af-a827-21f330507d39_isrForCrosstalkSources_1125943_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/413baf6e-cfe2-41af-a827-21f330507d39_isrForCrosstalkSources_1125943_58.stderr parsl.dataflow.dflow INFO: Task 1272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 375ddd9f501217a4b1e2e49f3cd23f16 with result from task 1272 parsl.dataflow.dflow DEBUG: Task 1272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/4975aa27-b14e-491d-8aba-beba6d3481c5_isrForCrosstalkSources_1125944_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/4975aa27-b14e-491d-8aba-beba6d3481c5_isrForCrosstalkSources_1125944_11.stderr parsl.dataflow.dflow INFO: Task 1279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2fcd971ec207385973e3c285321e761 with result from task 1279 parsl.dataflow.dflow DEBUG: Task 1279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/2c16d3a2-4fbf-43fd-b024-3be17a4f0adf_isrForCrosstalkSources_1125889_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/2c16d3a2-4fbf-43fd-b024-3be17a4f0adf_isrForCrosstalkSources_1125889_34.stderr parsl.dataflow.dflow INFO: Task 1278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb0d3112de25f4ac2b0436a9543be212 with result from task 1278 parsl.dataflow.dflow DEBUG: Task 1278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/4587f0a3-6d09-4258-a6ae-431ecd7ff800_isrForCrosstalkSources_1125906_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/4587f0a3-6d09-4258-a6ae-431ecd7ff800_isrForCrosstalkSources_1125906_14.stderr parsl.dataflow.dflow INFO: Task 1276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc29648440cb321ff3ae9ec98cd81c51 with result from task 1276 parsl.dataflow.dflow DEBUG: Task 1276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/ff008800-7982-4ad8-8956-59c4aeead3c5_isrForCrosstalkSources_1125934_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/ff008800-7982-4ad8-8956-59c4aeead3c5_isrForCrosstalkSources_1125934_44.stderr parsl.dataflow.dflow INFO: Task 1277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 921d092812d6ecc07633fba9f7c8cd4f with result from task 1277 parsl.dataflow.dflow DEBUG: Task 1277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/ded33ae2-c238-409b-8189-98fa213185b7_isrForCrosstalkSources_1125908_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/ded33ae2-c238-409b-8189-98fa213185b7_isrForCrosstalkSources_1125908_31.stderr parsl.dataflow.dflow INFO: Task 1275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c45d0220f690082b8e980de44380235 with result from task 1275 parsl.dataflow.dflow DEBUG: Task 1275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/9056c9d5-b812-4241-ba39-3d9ada7d7895_isrForCrosstalkSources_1125917_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/9056c9d5-b812-4241-ba39-3d9ada7d7895_isrForCrosstalkSources_1125917_22.stderr parsl.dataflow.dflow INFO: Task 1273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a55c0b2f20b2565356b3f1f7d3f1a814 with result from task 1273 parsl.dataflow.dflow DEBUG: Task 1273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/409eb1b6-ddce-477a-9b51-f148330f4ede_isrForCrosstalkSources_1125942_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/409eb1b6-ddce-477a-9b51-f148330f4ede_isrForCrosstalkSources_1125942_28.stderr parsl.dataflow.dflow INFO: Task 1262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 816552478c4e8b56e568f5e4ae621f0f with result from task 1262 parsl.dataflow.dflow DEBUG: Task 1262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/feafcb43-9ba4-45ff-813f-ca3c20eafa61_isrForCrosstalkSources_1125892_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/feafcb43-9ba4-45ff-813f-ca3c20eafa61_isrForCrosstalkSources_1125892_21.stderr parsl.dataflow.dflow INFO: Task 1263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea219e6d57db26340d164ce5d9150d7a with result from task 1263 parsl.dataflow.dflow DEBUG: Task 1263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/95bf0aad-63a2-4998-93c2-7dde14b09db9_isrForCrosstalkSources_1125909_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/95bf0aad-63a2-4998-93c2-7dde14b09db9_isrForCrosstalkSources_1125909_25.stderr parsl.dataflow.dflow INFO: Task 1258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9236b725ce2b94d44fece75c13e34957 with result from task 1258 parsl.dataflow.dflow DEBUG: Task 1258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/83d4c87e-3c5c-48ca-8fba-ff63cc17c41c_isrForCrosstalkSources_1125893_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/83d4c87e-3c5c-48ca-8fba-ff63cc17c41c_isrForCrosstalkSources_1125893_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:01 proc_lsst.multi:146 [INFO] found job 16896590 in provider checkpoint proc_lsst.multi INFO: found job 16896590 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a26b7839087c5d2368e00ed780853bdf with result from task 1529 parsl.dataflow.dflow DEBUG: Task 1529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/bcde2489-10eb-4a91-a8e9-cdd72c9c7824_isrForCrosstalkSources_1125911_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/bcde2489-10eb-4a91-a8e9-cdd72c9c7824_isrForCrosstalkSources_1125911_2.stderr parsl.dataflow.dflow INFO: Task 1222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9bec00d5e8e6f2964e3eac1fc27f7fc with result from task 1222 parsl.dataflow.dflow DEBUG: Task 1222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/f7832ebc-6791-4730-b321-5427e65a7088_isrForCrosstalkSources_1125900_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/f7832ebc-6791-4730-b321-5427e65a7088_isrForCrosstalkSources_1125900_24.stderr parsl.dataflow.dflow INFO: Task 1223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c18950b4805a3497e662276aae671cac with result from task 1223 parsl.dataflow.dflow DEBUG: Task 1223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/8e883939-750c-45ec-a781-b0f03eb3bac3_isrForCrosstalkSources_1125912_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/8e883939-750c-45ec-a781-b0f03eb3bac3_isrForCrosstalkSources_1125912_39.stderr parsl.dataflow.dflow INFO: Task 1224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffb99fc84382666c91a12ba5bdeffd8f with result from task 1224 parsl.dataflow.dflow DEBUG: Task 1224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/32ef0c89-4387-4648-97e5-938f6ab397f8_isrForCrosstalkSources_1125924_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/32ef0c89-4387-4648-97e5-938f6ab397f8_isrForCrosstalkSources_1125924_37.stderr parsl.dataflow.dflow INFO: Task 1340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d233b85bb9bde7a534cd87c06f102f3a with result from task 1340 parsl.dataflow.dflow DEBUG: Task 1340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/148b07a1-ffac-4cb9-bd83-aa98ea23c58d_isrForCrosstalkSources_1125914_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/148b07a1-ffac-4cb9-bd83-aa98ea23c58d_isrForCrosstalkSources_1125914_17.stderr parsl.dataflow.dflow INFO: Task 1345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bd5b0d9225d46606eecc8f19dc207da with result from task 1345 parsl.dataflow.dflow DEBUG: Task 1345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/a5daaa78-ce74-4ca7-8175-0a80a3f46602_isrForCrosstalkSources_1125907_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/a5daaa78-ce74-4ca7-8175-0a80a3f46602_isrForCrosstalkSources_1125907_24.stderr parsl.dataflow.dflow INFO: Task 1234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8c2df9d31c7a409768fda143144f6e0 with result from task 1234 parsl.dataflow.dflow DEBUG: Task 1234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/4a7ec24d-1f1c-4db9-b2b2-5d77bb147c86_isrForCrosstalkSources_1125942_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/4a7ec24d-1f1c-4db9-b2b2-5d77bb147c86_isrForCrosstalkSources_1125942_27.stderr parsl.dataflow.dflow INFO: Task 1235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4c0c4efd77af3012ab604417129f247 with result from task 1235 parsl.dataflow.dflow DEBUG: Task 1235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/b592ffee-ef4f-499c-80f3-cb8e2c565d57_isrForCrosstalkSources_1125901_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/b592ffee-ef4f-499c-80f3-cb8e2c565d57_isrForCrosstalkSources_1125901_1.stderr parsl.dataflow.dflow INFO: Task 1236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4372176920d991de4ac5622c8e30c13 with result from task 1236 parsl.dataflow.dflow DEBUG: Task 1236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/ae0f9712-999c-477f-96a0-8933821d2e42_isrForCrosstalkSources_1125910_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/ae0f9712-999c-477f-96a0-8933821d2e42_isrForCrosstalkSources_1125910_34.stderr parsl.dataflow.dflow INFO: Task 1237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7877e8feb3d23be4041858962f856291 with result from task 1237 parsl.dataflow.dflow DEBUG: Task 1237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/a7ee5a15-3980-4212-aa16-49171e59f9ac_isrForCrosstalkSources_1125897_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/a7ee5a15-3980-4212-aa16-49171e59f9ac_isrForCrosstalkSources_1125897_4.stderr parsl.dataflow.dflow INFO: Task 1231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 856931630f5e854bfcbe207bc0b7cf1d with result from task 1231 parsl.dataflow.dflow DEBUG: Task 1231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/e5803787-90ee-4da2-be94-d5d6218c1d0f_isrForCrosstalkSources_1125934_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/e5803787-90ee-4da2-be94-d5d6218c1d0f_isrForCrosstalkSources_1125934_15.stderr parsl.dataflow.dflow INFO: Task 1242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f37323c16b92ef65487cdc1e6c848bb with result from task 1242 parsl.dataflow.dflow DEBUG: Task 1242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/a11b7ffd-a296-454d-b9bf-610a311dbfde_isrForCrosstalkSources_1125908_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/a11b7ffd-a296-454d-b9bf-610a311dbfde_isrForCrosstalkSources_1125908_13.stderr parsl.dataflow.dflow INFO: Task 1243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 124601488abe5ae34462fecb31be2914 with result from task 1243 parsl.dataflow.dflow DEBUG: Task 1243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/ab560e94-3483-48a7-bb2f-509fb421c510_isrForCrosstalkSources_1125907_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/ab560e94-3483-48a7-bb2f-509fb421c510_isrForCrosstalkSources_1125907_13.stderr parsl.dataflow.dflow INFO: Task 1246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 090b2af441a6d5ece9b9c3f1bb48bca9 with result from task 1246 parsl.dataflow.dflow DEBUG: Task 1246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/4e6bca42-9488-4281-a1d0-bc45b877565e_isrForCrosstalkSources_1125919_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/4e6bca42-9488-4281-a1d0-bc45b877565e_isrForCrosstalkSources_1125919_32.stderr parsl.dataflow.dflow INFO: Task 1240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1929c2297f068375d5cf7daf0b350965 with result from task 1240 parsl.dataflow.dflow DEBUG: Task 1240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8224e884-ce2a-4976-98c7-00fb4b3efe99_isrForCrosstalkSources_1125944_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8224e884-ce2a-4976-98c7-00fb4b3efe99_isrForCrosstalkSources_1125944_23.stderr parsl.dataflow.dflow INFO: Task 1220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6af6f66425737ad471fb75862e7ae7b3 with result from task 1220 parsl.dataflow.dflow DEBUG: Task 1220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/266c5e34-15ff-4f49-9627-8044efa54ff3_isrForCrosstalkSources_1125942_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/266c5e34-15ff-4f49-9627-8044efa54ff3_isrForCrosstalkSources_1125942_32.stderr parsl.dataflow.dflow INFO: Task 1218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a22c6eadf247006b53c9492a2d503d1 with result from task 1218 parsl.dataflow.dflow DEBUG: Task 1218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/c1e3128c-3127-4c4b-9be2-2da8dbd7a7b7_isrForCrosstalkSources_1125899_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/c1e3128c-3127-4c4b-9be2-2da8dbd7a7b7_isrForCrosstalkSources_1125899_2.stderr parsl.dataflow.dflow INFO: Task 1219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69cd6c249ed8c2b9afc1d343dae7e345 with result from task 1219 parsl.dataflow.dflow DEBUG: Task 1219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/90ca9050-8730-4486-b051-c790003365ea_isrForCrosstalkSources_1125917_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/90ca9050-8730-4486-b051-c790003365ea_isrForCrosstalkSources_1125917_39.stderr parsl.dataflow.dflow INFO: Task 1544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 574351a7d25cbaafe4b646278583088d with result from task 1544 parsl.dataflow.dflow DEBUG: Task 1544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/43db868b-327b-4390-9f67-af0660193070_isrForCrosstalkSources_1125919_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/43db868b-327b-4390-9f67-af0660193070_isrForCrosstalkSources_1125919_52.stderr parsl.dataflow.dflow INFO: Task 1341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff5db0b3bf6318c11139dc25ef75a6d3 with result from task 1341 parsl.dataflow.dflow DEBUG: Task 1341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/f4182646-63ca-44af-9278-5ae7082585a8_isrForCrosstalkSources_1125925_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/f4182646-63ca-44af-9278-5ae7082585a8_isrForCrosstalkSources_1125925_13.stderr parsl.dataflow.dflow INFO: Task 1343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 602c885d2bb6788e926d3b66a9289de2 with result from task 1343 parsl.dataflow.dflow DEBUG: Task 1343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/46259f0f-a2db-4255-b326-b466a232f8a3_isrForCrosstalkSources_1125942_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/46259f0f-a2db-4255-b326-b466a232f8a3_isrForCrosstalkSources_1125942_9.stderr parsl.dataflow.dflow INFO: Task 1346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10f7132a985eddc5f34ea5c8dceb68e9 with result from task 1346 parsl.dataflow.dflow DEBUG: Task 1346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/4bd815d1-d280-4cff-922e-2de196b9e35f_isrForCrosstalkSources_1125939_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/4bd815d1-d280-4cff-922e-2de196b9e35f_isrForCrosstalkSources_1125939_54.stderr parsl.dataflow.dflow INFO: Task 1344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65720b8695c6f5c99c3d6841d4aa8875 with result from task 1344 parsl.dataflow.dflow DEBUG: Task 1344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/e8930649-0e10-4ff8-a775-a34741b0ab65_isrForCrosstalkSources_1125898_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/e8930649-0e10-4ff8-a775-a34741b0ab65_isrForCrosstalkSources_1125898_55.stderr parsl.dataflow.dflow INFO: Task 1180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff5322b7de89ad82d2db85c778676db7 with result from task 1180 parsl.dataflow.dflow DEBUG: Task 1180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/d0a6cf8d-698a-41a3-a325-5b59002ddcb4_isrForCrosstalkSources_1125938_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/d0a6cf8d-698a-41a3-a325-5b59002ddcb4_isrForCrosstalkSources_1125938_52.stderr parsl.dataflow.dflow INFO: Task 1225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6200ba5ebd9205883826bd3f6bcdae22 with result from task 1225 parsl.dataflow.dflow DEBUG: Task 1225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/1fea3c3c-a2bb-43ee-bfa2-c140dc43aad1_isrForCrosstalkSources_1125892_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/1fea3c3c-a2bb-43ee-bfa2-c140dc43aad1_isrForCrosstalkSources_1125892_26.stderr parsl.dataflow.dflow INFO: Task 1197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cac8509ce70d66f5ba6a31d62194c145 with result from task 1197 parsl.dataflow.dflow DEBUG: Task 1197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/44dc33a6-1119-435c-afc7-c95502ee2f2e_isrForCrosstalkSources_1125931_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/44dc33a6-1119-435c-afc7-c95502ee2f2e_isrForCrosstalkSources_1125931_46.stderr parsl.dataflow.dflow INFO: Task 1190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcf14cf1fe976aacbbfd364a16717476 with result from task 1190 parsl.dataflow.dflow DEBUG: Task 1190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/866cab21-8d68-48e6-a2d9-7e5c12f262c5_isrForCrosstalkSources_1125899_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/866cab21-8d68-48e6-a2d9-7e5c12f262c5_isrForCrosstalkSources_1125899_45.stderr parsl.dataflow.dflow INFO: Task 1217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 814fe8cc0f874ec3bf7e9cacdba3230a with result from task 1217 parsl.dataflow.dflow DEBUG: Task 1217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/1ecf97f4-1b70-4982-bada-6aefb1531e26_isrForCrosstalkSources_1125896_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/1ecf97f4-1b70-4982-bada-6aefb1531e26_isrForCrosstalkSources_1125896_10.stderr parsl.dataflow.dflow INFO: Task 1216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8603cc14eb708963f31cd2e3f9fbb9c with result from task 1216 parsl.dataflow.dflow DEBUG: Task 1216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/7406cf65-e0f6-455d-baae-1fd2f988bf37_isrForCrosstalkSources_1125889_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/7406cf65-e0f6-455d-baae-1fd2f988bf37_isrForCrosstalkSources_1125889_1.stderr parsl.dataflow.dflow INFO: Task 1198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3eaccee0fdb811de579789530637eaa6 with result from task 1198 parsl.dataflow.dflow DEBUG: Task 1198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/10ac8086-7eca-4b69-bad4-14fbb5250ed1_isrForCrosstalkSources_1125907_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/10ac8086-7eca-4b69-bad4-14fbb5250ed1_isrForCrosstalkSources_1125907_5.stderr parsl.dataflow.dflow INFO: Task 1203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23070d2df6230d4923627d86c48edf60 with result from task 1203 parsl.dataflow.dflow DEBUG: Task 1203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/17e38d45-3679-4060-8935-3127d9f38724_isrForCrosstalkSources_1125913_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/17e38d45-3679-4060-8935-3127d9f38724_isrForCrosstalkSources_1125913_47.stderr parsl.dataflow.dflow INFO: Task 1200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad9b1806188eef4c837856bc28a79e44 with result from task 1200 parsl.dataflow.dflow DEBUG: Task 1200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/90cf62bf-1145-40cf-9e35-578c928fa387_isrForCrosstalkSources_1125942_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/90cf62bf-1145-40cf-9e35-578c928fa387_isrForCrosstalkSources_1125942_12.stderr parsl.dataflow.dflow INFO: Task 1166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca690a2d4428a75bb407355eed97c4d4 with result from task 1166 parsl.dataflow.dflow DEBUG: Task 1166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2e3a2f2f-65d8-4c10-9c99-230e4e56094b_isrForCrosstalkSources_1125895_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2e3a2f2f-65d8-4c10-9c99-230e4e56094b_isrForCrosstalkSources_1125895_24.stderr parsl.dataflow.dflow INFO: Task 1169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69b58b94d90457ab2fa4a4df697911b3 with result from task 1169 parsl.dataflow.dflow DEBUG: Task 1169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/a037c7b3-d473-4177-bc25-489fa9aade01_isrForCrosstalkSources_1125896_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/a037c7b3-d473-4177-bc25-489fa9aade01_isrForCrosstalkSources_1125896_21.stderr parsl.dataflow.dflow INFO: Task 1187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa5008c2586d5d12bafd7767e3c11614 with result from task 1187 parsl.dataflow.dflow DEBUG: Task 1187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a22f7ed7-2864-4a00-b969-1fa5a58519e7_isrForCrosstalkSources_1125938_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a22f7ed7-2864-4a00-b969-1fa5a58519e7_isrForCrosstalkSources_1125938_35.stderr parsl.dataflow.dflow INFO: Task 1178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 267d8304a485d936f2e86124bc13f90d with result from task 1178 parsl.dataflow.dflow DEBUG: Task 1178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/3cb4f70f-7b88-42b2-b5d4-80bfed94d45b_isrForCrosstalkSources_1125898_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/3cb4f70f-7b88-42b2-b5d4-80bfed94d45b_isrForCrosstalkSources_1125898_12.stderr parsl.dataflow.dflow INFO: Task 1192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bead73593eaee5fcdd2e071ab470fc6 with result from task 1192 parsl.dataflow.dflow DEBUG: Task 1192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/7b9874f5-162d-4bef-bd34-218c3ae10a7f_isrForCrosstalkSources_1125946_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/7b9874f5-162d-4bef-bd34-218c3ae10a7f_isrForCrosstalkSources_1125946_11.stderr parsl.dataflow.dflow INFO: Task 1193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f70270ac3625c8967cb824a4f82658b with result from task 1193 parsl.dataflow.dflow DEBUG: Task 1193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/02c024c9-f192-4c85-aa27-527ad6ac65ed_isrForCrosstalkSources_1125922_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/02c024c9-f192-4c85-aa27-527ad6ac65ed_isrForCrosstalkSources_1125922_16.stderr parsl.dataflow.dflow INFO: Task 1196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a776e27d7b8bb565473b8e7dc80be847 with result from task 1196 parsl.dataflow.dflow DEBUG: Task 1196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/a65455f7-b887-4543-bf08-e8c32b3fffdc_isrForCrosstalkSources_1125905_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/a65455f7-b887-4543-bf08-e8c32b3fffdc_isrForCrosstalkSources_1125905_9.stderr parsl.dataflow.dflow INFO: Task 1195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f63a072b192dff6c4e9db4c666e21798 with result from task 1195 parsl.dataflow.dflow DEBUG: Task 1195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/6a679216-6d4c-465c-84a3-a9d377373723_isrForCrosstalkSources_1125900_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/6a679216-6d4c-465c-84a3-a9d377373723_isrForCrosstalkSources_1125900_59.stderr parsl.dataflow.dflow INFO: Task 1191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f913efd471917dfb9dfd4affa36b1e3 with result from task 1191 parsl.dataflow.dflow DEBUG: Task 1191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/64c15eb7-7ab9-42e7-acd2-605d72c472c6_isrForCrosstalkSources_1125923_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/64c15eb7-7ab9-42e7-acd2-605d72c472c6_isrForCrosstalkSources_1125923_16.stderr parsl.dataflow.dflow INFO: Task 1205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 302da1d4898426b49eea00e33f8daef4 with result from task 1205 parsl.dataflow.dflow DEBUG: Task 1205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f168732f-6c1f-41fb-8f1c-0c336be3fa1c_isrForCrosstalkSources_1125930_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f168732f-6c1f-41fb-8f1c-0c336be3fa1c_isrForCrosstalkSources_1125930_22.stderr parsl.dataflow.dflow INFO: Task 1199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1eef54b7a0c0b4cee7de145f0789243 with result from task 1199 parsl.dataflow.dflow DEBUG: Task 1199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/17a7d32e-65c0-4892-a591-6d143e3b458c_isrForCrosstalkSources_1125934_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/17a7d32e-65c0-4892-a591-6d143e3b458c_isrForCrosstalkSources_1125934_37.stderr parsl.dataflow.dflow INFO: Task 1202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b3e78028aad8e54f85188623643c9da with result from task 1202 parsl.dataflow.dflow DEBUG: Task 1202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/3d760f5f-484b-48b8-8312-9f84ffa2c459_isrForCrosstalkSources_1125892_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/3d760f5f-484b-48b8-8312-9f84ffa2c459_isrForCrosstalkSources_1125892_39.stderr parsl.dataflow.dflow INFO: Task 1201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 340c0b97d0f40d9ab62cb337611312ca with result from task 1201 parsl.dataflow.dflow DEBUG: Task 1201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/1325b789-00a3-4827-a58a-2133227148e9_isrForCrosstalkSources_1125898_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/1325b789-00a3-4827-a58a-2133227148e9_isrForCrosstalkSources_1125898_18.stderr parsl.dataflow.dflow INFO: Task 1167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57c4c8b9788d6b9ee8ec827b52e5b545 with result from task 1167 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:01 proc_lsst.multi:146 [INFO] found job 16896591 in provider checkpoint proc_lsst.multi INFO: found job 16896591 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/84039217-2f19-4512-bb6a-0c9e9853812b_isrForCrosstalkSources_1125939_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/84039217-2f19-4512-bb6a-0c9e9853812b_isrForCrosstalkSources_1125939_7.stderr parsl.dataflow.dflow INFO: Task 1165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7d74e0ff59ac158e231501f0d13b4c9 with result from task 1165 parsl.dataflow.dflow DEBUG: Task 1165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/2e4232ee-70b3-4216-ac1d-82edb063591a_isrForCrosstalkSources_1125908_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/2e4232ee-70b3-4216-ac1d-82edb063591a_isrForCrosstalkSources_1125908_10.stderr parsl.dataflow.dflow INFO: Task 1183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ebf8b8c5a6cad6947d1f8c5a0727dea with result from task 1183 parsl.dataflow.dflow DEBUG: Task 1183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/635c9963-df1e-452e-9f3a-c15475c16082_isrForCrosstalkSources_1125916_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/635c9963-df1e-452e-9f3a-c15475c16082_isrForCrosstalkSources_1125916_9.stderr parsl.dataflow.dflow INFO: Task 1181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bef3385975b1fd41daaf7a7f4c21def8 with result from task 1181 parsl.dataflow.dflow DEBUG: Task 1181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/4c9dbf7b-4baf-452a-8f1f-3aaebd35609e_isrForCrosstalkSources_1125923_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/4c9dbf7b-4baf-452a-8f1f-3aaebd35609e_isrForCrosstalkSources_1125923_4.stderr parsl.dataflow.dflow INFO: Task 1185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4eb2b0d604f0c558f28ab5aebbb2f380 with result from task 1185 parsl.dataflow.dflow DEBUG: Task 1185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d627eb0a-ecb8-4f73-90dd-ba11e183ee3f_isrForCrosstalkSources_1125919_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d627eb0a-ecb8-4f73-90dd-ba11e183ee3f_isrForCrosstalkSources_1125919_21.stderr parsl.dataflow.dflow INFO: Task 1174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eed8d30e388775d6751d72681b1d96dc with result from task 1174 parsl.dataflow.dflow DEBUG: Task 1174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/0e390301-90f8-4fe5-977c-9ec960a30abc_isrForCrosstalkSources_1125897_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/0e390301-90f8-4fe5-977c-9ec960a30abc_isrForCrosstalkSources_1125897_3.stderr parsl.dataflow.dflow INFO: Task 1175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0228719dd3ea4930f111ad91f21e5f96 with result from task 1175 parsl.dataflow.dflow DEBUG: Task 1175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/23273230-27ed-4631-882d-e6fc17daf858_isrForCrosstalkSources_1125931_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/23273230-27ed-4631-882d-e6fc17daf858_isrForCrosstalkSources_1125931_19.stderr parsl.dataflow.dflow INFO: Task 1176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fe8dce10349ec267e9ef3ff0cc908ae with result from task 1176 parsl.dataflow.dflow DEBUG: Task 1176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/69032172-33ca-43a6-8b22-0650d1a46ca5_isrForCrosstalkSources_1125923_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/69032172-33ca-43a6-8b22-0650d1a46ca5_isrForCrosstalkSources_1125923_32.stderr parsl.dataflow.dflow INFO: Task 1173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e561dfb9eb3c13131f2a422627a55346 with result from task 1173 parsl.dataflow.dflow DEBUG: Task 1173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/e7246c9b-deff-4b77-ad84-60107b6d6df4_isrForCrosstalkSources_1125919_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/e7246c9b-deff-4b77-ad84-60107b6d6df4_isrForCrosstalkSources_1125919_33.stderr parsl.dataflow.dflow INFO: Task 1177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83948121f3d34950e567120fcf871077 with result from task 1177 parsl.dataflow.dflow DEBUG: Task 1177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/746a7d7b-f23a-4c15-aac7-f09771376933_isrForCrosstalkSources_1125908_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/746a7d7b-f23a-4c15-aac7-f09771376933_isrForCrosstalkSources_1125908_56.stderr parsl.dataflow.dflow INFO: Task 1204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 042ca2d5606abd6bfafd3856f8cb25bc with result from task 1204 parsl.dataflow.dflow DEBUG: Task 1204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/cbfa505c-0dd4-4087-9137-7f08f52045c2_isrForCrosstalkSources_1125897_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/cbfa505c-0dd4-4087-9137-7f08f52045c2_isrForCrosstalkSources_1125897_22.stderr parsl.dataflow.dflow INFO: Task 1194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b98120856494eb1b80dc556c7f474e06 with result from task 1194 parsl.dataflow.dflow DEBUG: Task 1194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/4b1f0ba1-81df-46d4-86ea-2e35373df36a_isrForCrosstalkSources_1125924_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/4b1f0ba1-81df-46d4-86ea-2e35373df36a_isrForCrosstalkSources_1125924_33.stderr parsl.dataflow.dflow INFO: Task 1188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c607ebd16b61361af1c95581fee0c342 with result from task 1188 parsl.dataflow.dflow DEBUG: Task 1188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/e0a61e85-52d5-45f4-8882-7beee6355a3a_isrForCrosstalkSources_1125913_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/e0a61e85-52d5-45f4-8882-7beee6355a3a_isrForCrosstalkSources_1125913_26.stderr parsl.dataflow.dflow INFO: Task 1184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92b0dbe2033464bda48be93b5f24403f with result from task 1184 parsl.dataflow.dflow DEBUG: Task 1184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/b380be05-ba8c-4bf7-8b76-f6be4d430a27_isrForCrosstalkSources_1125909_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/b380be05-ba8c-4bf7-8b76-f6be4d430a27_isrForCrosstalkSources_1125909_13.stderr parsl.dataflow.dflow INFO: Task 1513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d0417b816cb14194bf7e0fd644caeee with result from task 1513 parsl.dataflow.dflow DEBUG: Task 1513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/d647a383-2ab0-4da0-b379-10b4260cff82_isrForCrosstalkSources_1125918_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/d647a383-2ab0-4da0-b379-10b4260cff82_isrForCrosstalkSources_1125918_6.stderr parsl.dataflow.dflow INFO: Task 1339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25542a0b517937cb77bea874212b5bbe with result from task 1339 parsl.dataflow.dflow DEBUG: Task 1339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/64b8f0ad-25dc-4c3d-ba57-7d658b1b2958_isrForCrosstalkSources_1125919_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/64b8f0ad-25dc-4c3d-ba57-7d658b1b2958_isrForCrosstalkSources_1125919_58.stderr parsl.dataflow.dflow INFO: Task 1577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78aa781ac17fc70e4e9a7af29dfa9adf with result from task 1577 parsl.dataflow.dflow DEBUG: Task 1577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/1fce1d9a-1224-4dcc-b9b4-ba18585ddeeb_isrForCrosstalkSources_1125936_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/1fce1d9a-1224-4dcc-b9b4-ba18585ddeeb_isrForCrosstalkSources_1125936_32.stderr parsl.dataflow.dflow INFO: Task 1249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9225e0d82848bf232b5759547a9e72e0 with result from task 1249 parsl.dataflow.dflow DEBUG: Task 1249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/b3eb25b2-1393-4c06-a739-dbc47ccbca1f_isrForCrosstalkSources_1125912_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/b3eb25b2-1393-4c06-a739-dbc47ccbca1f_isrForCrosstalkSources_1125912_8.stderr parsl.dataflow.dflow INFO: Task 1256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42d5089a9cbb54d4729471adf7e11f74 with result from task 1256 parsl.dataflow.dflow DEBUG: Task 1256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/45358667-cc37-486e-8948-8e8d593ac70a_isrForCrosstalkSources_1125936_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/45358667-cc37-486e-8948-8e8d593ac70a_isrForCrosstalkSources_1125936_55.stderr parsl.dataflow.dflow INFO: Task 1534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ebff62b786acc1610e5cbe2d8cec3b1 with result from task 1534 parsl.dataflow.dflow DEBUG: Task 1534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/76aaf5a3-195b-46e6-83ba-56bdd165959f_isrForCrosstalkSources_1125889_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/76aaf5a3-195b-46e6-83ba-56bdd165959f_isrForCrosstalkSources_1125889_60.stderr parsl.dataflow.dflow INFO: Task 1556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 345ecc2ff667085245cbd46f777fc599 with result from task 1556 parsl.dataflow.dflow DEBUG: Task 1556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/3c53ba15-795c-4375-bad6-80d74a365160_isrForCrosstalkSources_1125926_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/3c53ba15-795c-4375-bad6-80d74a365160_isrForCrosstalkSources_1125926_17.stderr parsl.dataflow.dflow INFO: Task 1227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3843d43206fe69a8108316bc73a87d7f with result from task 1227 parsl.dataflow.dflow DEBUG: Task 1227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/59bae909-03c1-4d56-987c-6e1ebc68ff85_isrForCrosstalkSources_1125938_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/59bae909-03c1-4d56-987c-6e1ebc68ff85_isrForCrosstalkSources_1125938_54.stderr parsl.dataflow.dflow INFO: Task 1361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1c7cb7e3a9707b7618f5a9817eee2c0 with result from task 1361 parsl.dataflow.dflow DEBUG: Task 1361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/764870d6-0313-494d-8c3b-5f3dbbdca103_isrForCrosstalkSources_1125914_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/764870d6-0313-494d-8c3b-5f3dbbdca103_isrForCrosstalkSources_1125914_27.stderr parsl.dataflow.dflow INFO: Task 1365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5db82598814d9b848ca4bc01d571e950 with result from task 1365 parsl.dataflow.dflow DEBUG: Task 1365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/93764c76-9d90-4375-ba55-a06edef6cfbd_isrForCrosstalkSources_1125937_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/93764c76-9d90-4375-ba55-a06edef6cfbd_isrForCrosstalkSources_1125937_19.stderr parsl.dataflow.dflow INFO: Task 1521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ebdf884bac3b04558da2592c2303b81 with result from task 1521 parsl.dataflow.dflow DEBUG: Task 1521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/2c88b2cb-1263-4853-96a3-848995052a27_isrForCrosstalkSources_1125940_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/2c88b2cb-1263-4853-96a3-848995052a27_isrForCrosstalkSources_1125940_3.stderr parsl.dataflow.dflow INFO: Task 1532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f509a5ab10e82bced4df3f0075980161 with result from task 1532 parsl.dataflow.dflow DEBUG: Task 1532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/bb78230d-2215-40a8-9d0d-ee11ef1e6778_isrForCrosstalkSources_1125892_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/bb78230d-2215-40a8-9d0d-ee11ef1e6778_isrForCrosstalkSources_1125892_8.stderr parsl.dataflow.dflow INFO: Task 1579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a72e40fd2e596a3ebe4cc4aea73aab82 with result from task 1579 parsl.dataflow.dflow DEBUG: Task 1579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/56c066d2-e8af-41fe-809e-e0176855bd52_isrForCrosstalkSources_1125934_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/56c066d2-e8af-41fe-809e-e0176855bd52_isrForCrosstalkSources_1125934_6.stderr parsl.dataflow.dflow INFO: Task 1519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8025bd30395e3d5a0d6fb637bb99b889 with result from task 1519 parsl.dataflow.dflow DEBUG: Task 1519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/9c7fd067-9c6f-4bb3-be36-35b875a9cf04_isrForCrosstalkSources_1125891_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/9c7fd067-9c6f-4bb3-be36-35b875a9cf04_isrForCrosstalkSources_1125891_58.stderr parsl.dataflow.dflow INFO: Task 1573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb4132d12a99fc979afd0d55c6c84c3d with result from task 1573 parsl.dataflow.dflow DEBUG: Task 1573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/60340e23-af89-4a70-af9b-ad280bbcd9a7_isrForCrosstalkSources_1125909_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/60340e23-af89-4a70-af9b-ad280bbcd9a7_isrForCrosstalkSources_1125909_6.stderr parsl.dataflow.dflow INFO: Task 1342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31d2fa39e9006e7ac73fa0605ee2bc4f with result from task 1342 parsl.dataflow.dflow DEBUG: Task 1342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/110cd67b-5008-4c26-93ba-42d165adea2a_isrForCrosstalkSources_1125912_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/110cd67b-5008-4c26-93ba-42d165adea2a_isrForCrosstalkSources_1125912_55.stderr parsl.dataflow.dflow INFO: Task 1215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e0578934f3fd50d8a0fd8fc874fc2f7 with result from task 1215 parsl.dataflow.dflow DEBUG: Task 1215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/80437f5a-51e8-4ebd-b1d8-baa3f2a6b433_isrForCrosstalkSources_1125889_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/80437f5a-51e8-4ebd-b1d8-baa3f2a6b433_isrForCrosstalkSources_1125889_2.stderr parsl.dataflow.dflow INFO: Task 1172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de871df7fb067199ce54a98f3a649afa with result from task 1172 parsl.dataflow.dflow DEBUG: Task 1172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/6ebe943a-9015-4801-b746-581445cb89c0_isrForCrosstalkSources_1125905_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/6ebe943a-9015-4801-b746-581445cb89c0_isrForCrosstalkSources_1125905_60.stderr parsl.dataflow.dflow INFO: Task 1171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b1ffc56a9efe140b6185b2b5a5a32bd with result from task 1171 parsl.dataflow.dflow DEBUG: Task 1171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/4a59661e-9aed-4fed-809e-0142e7acea6a_isrForCrosstalkSources_1125943_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/4a59661e-9aed-4fed-809e-0142e7acea6a_isrForCrosstalkSources_1125943_52.stderr parsl.dataflow.dflow INFO: Task 1578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffefd6eae6215386a96caa4a59bed82f with result from task 1578 parsl.dataflow.dflow DEBUG: Task 1578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/0d419855-baf6-4c77-939f-ade1fc5d699b_isrForCrosstalkSources_1125936_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/0d419855-baf6-4c77-939f-ade1fc5d699b_isrForCrosstalkSources_1125936_41.stderr parsl.dataflow.dflow INFO: Task 1179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ddb008ef3dbe8be09abdfd8569a2e3d with result from task 1179 parsl.dataflow.dflow DEBUG: Task 1179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/92017e0f-7d4f-4c56-a721-bcb656324728_isrForCrosstalkSources_1125935_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/92017e0f-7d4f-4c56-a721-bcb656324728_isrForCrosstalkSources_1125935_52.stderr parsl.dataflow.dflow INFO: Task 1168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd5bcba7c8f6f03921f172b538553f7e with result from task 1168 parsl.dataflow.dflow DEBUG: Task 1168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/c2766d34-c785-4807-84ac-59beb6612976_isrForCrosstalkSources_1125923_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/c2766d34-c785-4807-84ac-59beb6612976_isrForCrosstalkSources_1125923_52.stderr parsl.dataflow.dflow INFO: Task 1182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85e3a9ab5e42fdcd8a6548397e758cfa with result from task 1182 parsl.dataflow.dflow DEBUG: Task 1182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/d6f9a53f-7abd-48c2-b808-211aa86fa3c3_isrForCrosstalkSources_1125901_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/d6f9a53f-7abd-48c2-b808-211aa86fa3c3_isrForCrosstalkSources_1125901_48.stderr parsl.dataflow.dflow INFO: Task 1316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 706d508c2a678b44f36a70329ddf561a with result from task 1316 parsl.dataflow.dflow DEBUG: Task 1316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/896c9626-b8b0-4d62-9dc9-a4c291d29ff3_isrForCrosstalkSources_1125917_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/896c9626-b8b0-4d62-9dc9-a4c291d29ff3_isrForCrosstalkSources_1125917_26.stderr parsl.dataflow.dflow INFO: Task 1313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1284a566ae8855c340e64b9cf67d49e9 with result from task 1313 parsl.dataflow.dflow DEBUG: Task 1313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/195deb70-0a45-4fcf-b84d-befc397b8c9d_isrForCrosstalkSources_1125947_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/195deb70-0a45-4fcf-b84d-befc397b8c9d_isrForCrosstalkSources_1125947_4.stderr parsl.dataflow.dflow INFO: Task 1567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 716f4e1371ab2d9ba42d1a0b219228ba with result from task 1567 parsl.dataflow.dflow DEBUG: Task 1567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/27c07e48-bd12-4300-b570-ec71f59bf526_isrForCrosstalkSources_1125903_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/27c07e48-bd12-4300-b570-ec71f59bf526_isrForCrosstalkSources_1125903_32.stderr parsl.dataflow.dflow INFO: Task 1536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61e4cdd6090da921aeca894db3b75ef9 with result from task 1536 parsl.dataflow.dflow DEBUG: Task 1536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/0d4b75ea-f584-49ea-8039-d988bb0a1da0_isrForCrosstalkSources_1125911_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/0d4b75ea-f584-49ea-8039-d988bb0a1da0_isrForCrosstalkSources_1125911_12.stderr parsl.dataflow.dflow INFO: Task 1232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae328fbd4f20f826931ecbc2ae7165de with result from task 1232 parsl.dataflow.dflow DEBUG: Task 1232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c1650ccb-71b3-43ce-a12c-e81da83fcbba_isrForCrosstalkSources_1125904_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c1650ccb-71b3-43ce-a12c-e81da83fcbba_isrForCrosstalkSources_1125904_6.stderr parsl.dataflow.dflow INFO: Task 1252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aea236503a45e2ace9c66d8245f8e0b6 with result from task 1252 parsl.dataflow.dflow DEBUG: Task 1252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/6b534939-c4c7-44ac-b61a-e990eb0a8508_isrForCrosstalkSources_1125913_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/6b534939-c4c7-44ac-b61a-e990eb0a8508_isrForCrosstalkSources_1125913_16.stderr parsl.dataflow.dflow INFO: Task 1308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a1d9337e320ce983baf1c4f1b3c803f with result from task 1308 parsl.dataflow.dflow DEBUG: Task 1308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/46800cc2-7aca-46bb-9703-c23cb06a5527_isrForCrosstalkSources_1125940_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/46800cc2-7aca-46bb-9703-c23cb06a5527_isrForCrosstalkSources_1125940_32.stderr parsl.dataflow.dflow INFO: Task 1295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f40fbaf61feeb9daaf67d2c95475adaa with result from task 1295 parsl.dataflow.dflow DEBUG: Task 1295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7bfcbab8-e1d0-4d30-87a8-c10ee8644157_isrForCrosstalkSources_1125929_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7bfcbab8-e1d0-4d30-87a8-c10ee8644157_isrForCrosstalkSources_1125929_1.stderr parsl.dataflow.dflow INFO: Task 1297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e619641032ee7f2892f03941e8b68d92 with result from task 1297 parsl.dataflow.dflow DEBUG: Task 1297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/23ed7ffb-ff1d-4ef2-9b27-14f977175360_isrForCrosstalkSources_1125900_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/23ed7ffb-ff1d-4ef2-9b27-14f977175360_isrForCrosstalkSources_1125900_32.stderr parsl.dataflow.dflow INFO: Task 1298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc304456c8bf05bf225f461d6263c39b with result from task 1298 parsl.dataflow.dflow DEBUG: Task 1298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/34e9d438-ab11-4dbb-92e7-316d09967d17_isrForCrosstalkSources_1125929_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/34e9d438-ab11-4dbb-92e7-316d09967d17_isrForCrosstalkSources_1125929_14.stderr parsl.dataflow.dflow INFO: Task 1301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 408d9ea08a8629f1b986d1182a6fd5d0 with result from task 1301 parsl.dataflow.dflow DEBUG: Task 1301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/a9534ca5-3b26-49d1-9372-6203a7b53c4a_isrForCrosstalkSources_1125929_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/a9534ca5-3b26-49d1-9372-6203a7b53c4a_isrForCrosstalkSources_1125929_22.stderr parsl.dataflow.dflow INFO: Task 1303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fda636459e37f616ca2b0218cd368aba with result from task 1303 parsl.dataflow.dflow DEBUG: Task 1303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/b572b8c6-58a5-496c-a52e-8d75acad1f72_isrForCrosstalkSources_1125940_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/b572b8c6-58a5-496c-a52e-8d75acad1f72_isrForCrosstalkSources_1125940_46.stderr parsl.dataflow.dflow INFO: Task 1238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f678a7e6bb0d71b575ba9fe898893a4 with result from task 1238 parsl.dataflow.dflow DEBUG: Task 1238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/6609779a-0470-4c02-baaf-b8d971f8aa3f_isrForCrosstalkSources_1125926_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/6609779a-0470-4c02-baaf-b8d971f8aa3f_isrForCrosstalkSources_1125926_7.stderr parsl.dataflow.dflow INFO: Task 1170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d43b61deae843e77c674997bf9d839fb with result from task 1170 parsl.dataflow.dflow DEBUG: Task 1170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/84744ce2-72fd-464b-82c1-64ca60c0776a_isrForCrosstalkSources_1125909_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/84744ce2-72fd-464b-82c1-64ca60c0776a_isrForCrosstalkSources_1125909_47.stderr parsl.dataflow.dflow INFO: Task 1230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ad24ed7336b43f5216457ff62cc15d0 with result from task 1230 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:03 proc_lsst.multi:146 [INFO] found job 16896592 in provider checkpoint proc_lsst.multi INFO: found job 16896592 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/5d02f9eb-c75b-4329-a745-caa690a2b0f1_isrForCrosstalkSources_1125932_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/5d02f9eb-c75b-4329-a745-caa690a2b0f1_isrForCrosstalkSources_1125932_36.stderr parsl.dataflow.dflow INFO: Task 1555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d488ffe3c92525bc935f9a6e65fc5272 with result from task 1555 parsl.dataflow.dflow DEBUG: Task 1555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/04b49bdb-7fb2-4831-b626-e1452dd22b8b_isrForCrosstalkSources_1125912_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/04b49bdb-7fb2-4831-b626-e1452dd22b8b_isrForCrosstalkSources_1125912_23.stderr parsl.dataflow.dflow INFO: Task 1389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 536c68c3d6f9834e929d59a1114822a6 with result from task 1389 parsl.dataflow.dflow DEBUG: Task 1389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/073f7a94-81c6-4c0c-9759-e987b478ab57_isrForCrosstalkSources_1125892_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/073f7a94-81c6-4c0c-9759-e987b478ab57_isrForCrosstalkSources_1125892_10.stderr parsl.dataflow.dflow INFO: Task 1396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 631c67fed881db80c4e5893b0003d34c with result from task 1396 parsl.dataflow.dflow DEBUG: Task 1396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/c9295d97-0a61-4381-8f26-f70f2891ef93_isrForCrosstalkSources_1125919_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/c9295d97-0a61-4381-8f26-f70f2891ef93_isrForCrosstalkSources_1125919_39.stderr parsl.dataflow.dflow INFO: Task 1392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 992d5f3f7dddc2169dc2a81bb8f6bd34 with result from task 1392 parsl.dataflow.dflow DEBUG: Task 1392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/a993ea55-174f-4c06-a240-e8d2c508ab66_isrForCrosstalkSources_1125890_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/a993ea55-174f-4c06-a240-e8d2c508ab66_isrForCrosstalkSources_1125890_39.stderr parsl.dataflow.dflow INFO: Task 1393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 994eda556e0473f0accff5f5ebf67af0 with result from task 1393 parsl.dataflow.dflow DEBUG: Task 1393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/00418d17-30f5-4ac1-a691-cdeccf6815b9_isrForCrosstalkSources_1125908_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/00418d17-30f5-4ac1-a691-cdeccf6815b9_isrForCrosstalkSources_1125908_20.stderr parsl.dataflow.dflow INFO: Task 1247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee21e1586044923bafaecbbd5f9188c2 with result from task 1247 parsl.dataflow.dflow DEBUG: Task 1247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/007f31f0-0b9b-4862-8a7c-89389763a76a_isrForCrosstalkSources_1125908_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/007f31f0-0b9b-4862-8a7c-89389763a76a_isrForCrosstalkSources_1125908_35.stderr parsl.dataflow.dflow INFO: Task 1568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4feef8c802afc2e6c1d0a9a1506b91d with result from task 1568 parsl.dataflow.dflow DEBUG: Task 1568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/fc5b27e6-fb09-4a00-850e-c1e408acbcfb_isrForCrosstalkSources_1125898_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/fc5b27e6-fb09-4a00-850e-c1e408acbcfb_isrForCrosstalkSources_1125898_2.stderr parsl.dataflow.dflow INFO: Task 1547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05854fb761c44ddbacb183c5ad805fbf with result from task 1547 parsl.dataflow.dflow DEBUG: Task 1547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/431202fc-ea44-414e-a1a6-23c93b8d627d_isrForCrosstalkSources_1125921_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/431202fc-ea44-414e-a1a6-23c93b8d627d_isrForCrosstalkSources_1125921_30.stderr parsl.dataflow.dflow INFO: Task 1376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddb4a4418522d850a136260dab2f0b07 with result from task 1376 parsl.dataflow.dflow DEBUG: Task 1376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/55379f22-c4bb-4334-a727-52538802d1e7_isrForCrosstalkSources_1125895_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/55379f22-c4bb-4334-a727-52538802d1e7_isrForCrosstalkSources_1125895_41.stderr parsl.dataflow.dflow INFO: Task 1380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f95c371db5338056336c070e76ed1ce with result from task 1380 parsl.dataflow.dflow DEBUG: Task 1380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/10aa9fc9-a8a6-4c8d-8370-25151756a733_isrForCrosstalkSources_1125909_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/10aa9fc9-a8a6-4c8d-8370-25151756a733_isrForCrosstalkSources_1125909_22.stderr parsl.dataflow.dflow INFO: Task 1229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 462f92d2cf6a34c830cf6541c0283219 with result from task 1229 parsl.dataflow.dflow DEBUG: Task 1229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/9e3a3f55-01d4-4911-9150-b09c245841a1_isrForCrosstalkSources_1125920_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/9e3a3f55-01d4-4911-9150-b09c245841a1_isrForCrosstalkSources_1125920_44.stderr parsl.dataflow.dflow INFO: Task 1254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef3548ce94b9a3ecf2e2a23f0889e372 with result from task 1254 parsl.dataflow.dflow DEBUG: Task 1254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/4e996b9c-2c3d-442a-8cf6-4d2a57652923_isrForCrosstalkSources_1125936_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/4e996b9c-2c3d-442a-8cf6-4d2a57652923_isrForCrosstalkSources_1125936_27.stderr parsl.dataflow.dflow INFO: Task 1359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35794e3a81fe8aae8cdb32a522ee1bc6 with result from task 1359 parsl.dataflow.dflow DEBUG: Task 1359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/01bce10b-6b78-4985-a874-fe0f84f75f5f_isrForCrosstalkSources_1125914_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/01bce10b-6b78-4985-a874-fe0f84f75f5f_isrForCrosstalkSources_1125914_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:03 proc_lsst.multi:146 [INFO] found job 16896593 in provider checkpoint proc_lsst.multi INFO: found job 16896593 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1e3a8dd8ce05c7f16e038df1c69a4d1 with result from task 1260 parsl.dataflow.dflow DEBUG: Task 1260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/da79051c-b6d7-4ccd-bfcf-0fcd8adba3ed_isrForCrosstalkSources_1125927_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/da79051c-b6d7-4ccd-bfcf-0fcd8adba3ed_isrForCrosstalkSources_1125927_55.stderr parsl.dataflow.dflow INFO: Task 1559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d4a02b7432abbcce8fcc3ef329fed87 with result from task 1559 parsl.dataflow.dflow DEBUG: Task 1559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/93075154-b142-4556-ab2d-fb00fb0fe242_isrForCrosstalkSources_1125915_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/93075154-b142-4556-ab2d-fb00fb0fe242_isrForCrosstalkSources_1125915_45.stderr parsl.dataflow.dflow INFO: Task 1378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1c683c923606bd0320960bdf10addef with result from task 1378 parsl.dataflow.dflow DEBUG: Task 1378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/aa34c9b0-ba58-4c80-a760-a1f0b57f8774_isrForCrosstalkSources_1125931_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/aa34c9b0-ba58-4c80-a760-a1f0b57f8774_isrForCrosstalkSources_1125931_45.stderr parsl.dataflow.dflow INFO: Task 1351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ac18d9c04a45277b86ae3ae86eb1f36 with result from task 1351 parsl.dataflow.dflow DEBUG: Task 1351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/26fe864f-4b6d-4d6b-826e-b1f9d6a22012_isrForCrosstalkSources_1125940_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/26fe864f-4b6d-4d6b-826e-b1f9d6a22012_isrForCrosstalkSources_1125940_51.stderr parsl.dataflow.dflow INFO: Task 1355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be26339edd73041bed0b63dfe3e346c2 with result from task 1355 parsl.dataflow.dflow DEBUG: Task 1355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/4d872d71-36bd-4eb5-a686-d4e2e5e395c0_isrForCrosstalkSources_1125921_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/4d872d71-36bd-4eb5-a686-d4e2e5e395c0_isrForCrosstalkSources_1125921_39.stderr parsl.dataflow.dflow INFO: Task 1356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e25956256174886226c78ce0df5b9b38 with result from task 1356 parsl.dataflow.dflow DEBUG: Task 1356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/eeb1eaac-34fb-4809-877c-9766f4082b02_isrForCrosstalkSources_1125893_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/eeb1eaac-34fb-4809-877c-9766f4082b02_isrForCrosstalkSources_1125893_18.stderr parsl.dataflow.dflow INFO: Task 1353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe6c58e6ef4c0e4b066af32c678dd5b6 with result from task 1353 parsl.dataflow.dflow DEBUG: Task 1353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/1a44c668-988d-40fa-866b-27b986eca3a1_isrForCrosstalkSources_1125893_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/1a44c668-988d-40fa-866b-27b986eca3a1_isrForCrosstalkSources_1125893_15.stderr parsl.dataflow.dflow INFO: Task 1575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79a9645a17d18b3ed010423978fc438d with result from task 1575 parsl.dataflow.dflow DEBUG: Task 1575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/9e89495d-ec54-4bc8-91d8-783b18974edf_isrForCrosstalkSources_1125928_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/9e89495d-ec54-4bc8-91d8-783b18974edf_isrForCrosstalkSources_1125928_19.stderr parsl.dataflow.dflow INFO: Task 1375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8587e28859857ad58614e80d4a5402ff with result from task 1375 parsl.dataflow.dflow DEBUG: Task 1375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/291e20e4-ee50-48be-9bea-53b97df8659f_isrForCrosstalkSources_1125920_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/291e20e4-ee50-48be-9bea-53b97df8659f_isrForCrosstalkSources_1125920_56.stderr parsl.dataflow.dflow INFO: Task 1379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe63abebec5bee86245db2bdd7a91184 with result from task 1379 parsl.dataflow.dflow DEBUG: Task 1379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/ce8f83ce-367f-4507-b98a-cbd602570333_isrForCrosstalkSources_1125929_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/ce8f83ce-367f-4507-b98a-cbd602570333_isrForCrosstalkSources_1125929_21.stderr parsl.dataflow.dflow INFO: Task 1570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9b9178ac805f1925a7613b1ccfb41fb with result from task 1570 parsl.dataflow.dflow DEBUG: Task 1570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e3e931af-468e-4073-9728-6a07e9fca96b_isrForCrosstalkSources_1125895_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e3e931af-468e-4073-9728-6a07e9fca96b_isrForCrosstalkSources_1125895_9.stderr parsl.dataflow.dflow INFO: Task 1349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54726f6e840d2888a7def361978c7616 with result from task 1349 parsl.dataflow.dflow DEBUG: Task 1349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/8907f801-e875-4a49-b80d-55c21d96d905_isrForCrosstalkSources_1125895_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/8907f801-e875-4a49-b80d-55c21d96d905_isrForCrosstalkSources_1125895_1.stderr parsl.dataflow.dflow INFO: Task 1350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b363289e807d29ca857f7273c6adb1b2 with result from task 1350 parsl.dataflow.dflow DEBUG: Task 1350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e5801472-6963-40df-b8b0-6158b1cbe440_isrForCrosstalkSources_1125895_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e5801472-6963-40df-b8b0-6158b1cbe440_isrForCrosstalkSources_1125895_46.stderr parsl.dataflow.dflow INFO: Task 1354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f9941fcb25a268ed4d4795fe64f6100 with result from task 1354 parsl.dataflow.dflow DEBUG: Task 1354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/a5f3129d-f7a4-4981-8e45-0e9e96e91f49_isrForCrosstalkSources_1125919_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/a5f3129d-f7a4-4981-8e45-0e9e96e91f49_isrForCrosstalkSources_1125919_55.stderr parsl.dataflow.dflow INFO: Task 1226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 445175aefe2a6366861ba92e84a5ec73 with result from task 1226 parsl.dataflow.dflow DEBUG: Task 1226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/0be87a6e-d4b5-4ed1-9ed0-a386b8f9b3c0_isrForCrosstalkSources_1125905_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/0be87a6e-d4b5-4ed1-9ed0-a386b8f9b3c0_isrForCrosstalkSources_1125905_53.stderr parsl.dataflow.dflow INFO: Task 1394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04afdfa82b8af72379644f1650bc9394 with result from task 1394 parsl.dataflow.dflow DEBUG: Task 1394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4ab5a2e9-3541-451e-be99-e6d632d882c2_isrForCrosstalkSources_1125908_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4ab5a2e9-3541-451e-be99-e6d632d882c2_isrForCrosstalkSources_1125908_3.stderr parsl.dataflow.dflow INFO: Task 1257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 989c472cd53157e11f10c8722c383b57 with result from task 1257 parsl.dataflow.dflow DEBUG: Task 1257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/8a596052-9723-4e25-b219-5db29ff75940_isrForCrosstalkSources_1125915_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/8a596052-9723-4e25-b219-5db29ff75940_isrForCrosstalkSources_1125915_60.stderr parsl.dataflow.dflow INFO: Task 1186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f784e82516e09ac66e8c0ebc558cc3f5 with result from task 1186 parsl.dataflow.dflow DEBUG: Task 1186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/969c9918-9a44-4d30-9f7e-35177d9e2066_isrForCrosstalkSources_1125906_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/969c9918-9a44-4d30-9f7e-35177d9e2066_isrForCrosstalkSources_1125906_60.stderr parsl.dataflow.dflow INFO: Task 1564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55608e5095a8c109e6dc0e5cf88f0b1d with result from task 1564 parsl.dataflow.dflow DEBUG: Task 1564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/342f472f-52b2-4320-858d-34eab12d541c_isrForCrosstalkSources_1125895_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/342f472f-52b2-4320-858d-34eab12d541c_isrForCrosstalkSources_1125895_5.stderr parsl.dataflow.dflow INFO: Task 1221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab795bc63d90aa05efa4cad1e1aed287 with result from task 1221 parsl.dataflow.dflow DEBUG: Task 1221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/5cca5be0-e49e-493e-a783-78f44cf1803d_isrForCrosstalkSources_1125903_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/5cca5be0-e49e-493e-a783-78f44cf1803d_isrForCrosstalkSources_1125903_55.stderr parsl.dataflow.dflow INFO: Task 1300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d30a18c82621759dad74533e7d7f7ad6 with result from task 1300 parsl.dataflow.dflow DEBUG: Task 1300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/8b2fc2ff-f68e-454d-83fc-24ef32de8074_isrForCrosstalkSources_1125914_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/8b2fc2ff-f68e-454d-83fc-24ef32de8074_isrForCrosstalkSources_1125914_15.stderr parsl.dataflow.dflow INFO: Task 1550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7d9086513a6f4a055ba4913f49d4e9c with result from task 1550 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:04 proc_lsst.multi:146 [INFO] found job 16896594 in provider checkpoint proc_lsst.multi INFO: found job 16896594 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c89347a9-f3d5-4a54-915d-4a4aba90ba37_isrForCrosstalkSources_1125917_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c89347a9-f3d5-4a54-915d-4a4aba90ba37_isrForCrosstalkSources_1125917_34.stderr parsl.dataflow.dflow INFO: Task 1309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ccc9044f38d91899e3dd9059c7cbd88 with result from task 1309 parsl.dataflow.dflow DEBUG: Task 1309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/2f282985-f0d4-4287-9409-e6f99e1516ce_isrForCrosstalkSources_1125910_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/2f282985-f0d4-4287-9409-e6f99e1516ce_isrForCrosstalkSources_1125910_17.stderr parsl.dataflow.dflow INFO: Task 1306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f61ee0249e376edd8bd47cc9047b9a4 with result from task 1306 parsl.dataflow.dflow DEBUG: Task 1306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d179dadc-4384-4141-9c1c-353ee47ca0cb_isrForCrosstalkSources_1125911_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d179dadc-4384-4141-9c1c-353ee47ca0cb_isrForCrosstalkSources_1125911_15.stderr parsl.dataflow.dflow INFO: Task 1241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f54d06ecefb4e33f371fdfcf6a7f10ce with result from task 1241 parsl.dataflow.dflow DEBUG: Task 1241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/525d2620-eb2e-4a7b-b3e8-22abdfc95680_isrForCrosstalkSources_1125947_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/525d2620-eb2e-4a7b-b3e8-22abdfc95680_isrForCrosstalkSources_1125947_52.stderr parsl.dataflow.dflow INFO: Task 1582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08e69473a9abb07ab89756c30ae3a5d3 with result from task 1582 parsl.dataflow.dflow DEBUG: Task 1582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/0fac113c-4c05-47c3-a247-ed8aec0a6f3e_isrForCrosstalkSources_1125918_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/0fac113c-4c05-47c3-a247-ed8aec0a6f3e_isrForCrosstalkSources_1125918_8.stderr parsl.dataflow.dflow INFO: Task 1565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry beebabc89e1b5ceea6def00ee905c3d3 with result from task 1565 parsl.dataflow.dflow DEBUG: Task 1565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/a819b7f0-e9ae-4d3e-929e-1b056c0d6311_isrForCrosstalkSources_1125916_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/a819b7f0-e9ae-4d3e-929e-1b056c0d6311_isrForCrosstalkSources_1125916_39.stderr parsl.dataflow.dflow INFO: Task 1560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 721f1b906ff0cd69e75eeb977dad5176 with result from task 1560 parsl.dataflow.dflow DEBUG: Task 1560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/8cf19534-b069-48f1-a7d6-6004df818572_isrForCrosstalkSources_1125927_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/8cf19534-b069-48f1-a7d6-6004df818572_isrForCrosstalkSources_1125927_60.stderr parsl.dataflow.dflow INFO: Task 1305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61d8604e2d8af102d66b777687b14468 with result from task 1305 parsl.dataflow.dflow DEBUG: Task 1305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/2c813077-3455-488a-882d-efec149be6ad_isrForCrosstalkSources_1125921_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/2c813077-3455-488a-882d-efec149be6ad_isrForCrosstalkSources_1125921_41.stderr parsl.dataflow.dflow INFO: Task 1261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fab7cfb5c223bbd4f0ef8f26f5cea26 with result from task 1261 parsl.dataflow.dflow DEBUG: Task 1261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/337d51a1-476b-4737-adfa-3bc77e9d30d0_isrForCrosstalkSources_1125924_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/337d51a1-476b-4737-adfa-3bc77e9d30d0_isrForCrosstalkSources_1125924_55.stderr parsl.dataflow.dflow INFO: Task 1557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b48ab2e935617724564536117989f5b with result from task 1557 parsl.dataflow.dflow DEBUG: Task 1557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/124fc6f9-5259-4d5d-b289-1b7bc539a22b_isrForCrosstalkSources_1125922_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/124fc6f9-5259-4d5d-b289-1b7bc539a22b_isrForCrosstalkSources_1125922_23.stderr parsl.dataflow.dflow INFO: Task 1304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8084b0e579730acc7c6c029e010a9414 with result from task 1304 parsl.dataflow.dflow DEBUG: Task 1304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/245a3ebc-1198-4ef0-a76a-9a7032ff337a_isrForCrosstalkSources_1125892_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/245a3ebc-1198-4ef0-a76a-9a7032ff337a_isrForCrosstalkSources_1125892_30.stderr parsl.dataflow.dflow INFO: Task 1381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f476bcc7d8e259fe4430e74695f02abd with result from task 1381 parsl.dataflow.dflow DEBUG: Task 1381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/79f214bc-c848-4b9d-9fa6-684aa1d42e9c_isrForCrosstalkSources_1125924_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/79f214bc-c848-4b9d-9fa6-684aa1d42e9c_isrForCrosstalkSources_1125924_2.stderr parsl.dataflow.dflow INFO: Task 1546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b78d9563795cf01c66524e51900a16d0 with result from task 1546 parsl.dataflow.dflow DEBUG: Task 1546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/e4d6fd2e-b8f8-4a46-b8a1-96f2bc77ba99_isrForCrosstalkSources_1125947_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/e4d6fd2e-b8f8-4a46-b8a1-96f2bc77ba99_isrForCrosstalkSources_1125947_49.stderr parsl.dataflow.dflow INFO: Task 1244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c13044e439f6ef99763bfed1a9aa9a46 with result from task 1244 parsl.dataflow.dflow DEBUG: Task 1244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/d7a2087e-fd24-4d30-85ba-87ac9053c0bf_isrForCrosstalkSources_1125903_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/d7a2087e-fd24-4d30-85ba-87ac9053c0bf_isrForCrosstalkSources_1125903_47.stderr parsl.dataflow.dflow INFO: Task 1549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aab00952b4bf71f768b875667e58f3df with result from task 1549 parsl.dataflow.dflow DEBUG: Task 1549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/55acf85f-9cfb-4c59-bcec-bd261ea08a2b_isrForCrosstalkSources_1125913_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/55acf85f-9cfb-4c59-bcec-bd261ea08a2b_isrForCrosstalkSources_1125913_5.stderr parsl.dataflow.dflow INFO: Task 1228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 156b69576449aa94b72777426cbaba52 with result from task 1228 parsl.dataflow.dflow DEBUG: Task 1228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/506568c6-ac2b-419a-aad5-a423f54e7d6b_isrForCrosstalkSources_1125927_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/506568c6-ac2b-419a-aad5-a423f54e7d6b_isrForCrosstalkSources_1125927_57.stderr parsl.dataflow.dflow INFO: Task 1552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 506816c8c9a1168e3829aad624a907d6 with result from task 1552 parsl.dataflow.dflow DEBUG: Task 1552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/e099f92d-722b-4803-a9fa-74fba2d14cc0_isrForCrosstalkSources_1125944_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/e099f92d-722b-4803-a9fa-74fba2d14cc0_isrForCrosstalkSources_1125944_53.stderr parsl.dataflow.dflow INFO: Task 1566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af7f4579a0fa862293c9f4da3a12ad71 with result from task 1566 parsl.dataflow.dflow DEBUG: Task 1566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/4376f5e2-88be-4980-8cf8-c9a5126d211a_isrForCrosstalkSources_1125904_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/4376f5e2-88be-4980-8cf8-c9a5126d211a_isrForCrosstalkSources_1125904_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 1233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bf4677b9d5c421161e95244644d187b with result from task 1233 parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:04 proc_lsst.multi:146 [INFO] found job 16896595 in provider checkpoint proc_lsst.multi INFO: found job 16896595 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/2f89dd3b-8364-484b-8fa7-1394f33447fa_isrForCrosstalkSources_1125893_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/2f89dd3b-8364-484b-8fa7-1394f33447fa_isrForCrosstalkSources_1125893_3.stderr parsl.dataflow.dflow INFO: Task 1326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f95bbf69a566054e2ed68703d3248c47 with result from task 1326 parsl.dataflow.dflow DEBUG: Task 1326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/427128e4-9b58-42ed-9599-2372ec1253cc_isrForCrosstalkSources_1125926_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/427128e4-9b58-42ed-9599-2372ec1253cc_isrForCrosstalkSources_1125926_19.stderr parsl.dataflow.dflow INFO: Task 1331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cfc8f296a76e26db5bee6208f8d08ee with result from task 1331 parsl.dataflow.dflow DEBUG: Task 1331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/28780cc2-6441-4e49-9530-dc725bb4d8fc_isrForCrosstalkSources_1125905_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/28780cc2-6441-4e49-9530-dc725bb4d8fc_isrForCrosstalkSources_1125905_19.stderr parsl.dataflow.dflow INFO: Task 1338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2fad23a5289e8b85263b0b5e0f6ffc7 with result from task 1338 parsl.dataflow.dflow DEBUG: Task 1338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/19de03a4-f313-4a7e-8535-b8e357a13763_isrForCrosstalkSources_1125940_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/19de03a4-f313-4a7e-8535-b8e357a13763_isrForCrosstalkSources_1125940_34.stderr parsl.dataflow.dflow INFO: Task 1360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fc15bcb56b5c98fcb4dcc1f87b4c19e with result from task 1360 parsl.dataflow.dflow DEBUG: Task 1360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/a0382ee6-9d22-4734-af4e-f39768ba91c0_isrForCrosstalkSources_1125930_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/a0382ee6-9d22-4734-af4e-f39768ba91c0_isrForCrosstalkSources_1125930_17.stderr parsl.dataflow.dflow INFO: Task 1553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33d731ab4b16bba2efb8fc4d7c5eece1 with result from task 1553 parsl.dataflow.dflow DEBUG: Task 1553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/bff942ce-6973-44e1-89f6-e1630d6a05da_isrForCrosstalkSources_1125902_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/bff942ce-6973-44e1-89f6-e1630d6a05da_isrForCrosstalkSources_1125902_38.stderr parsl.dataflow.dflow INFO: Task 1377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14a791315b65319a64e8cea444488fc2 with result from task 1377 parsl.dataflow.dflow DEBUG: Task 1377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/0245f702-bb85-4234-84e7-93612e7cf9fe_isrForCrosstalkSources_1125925_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/0245f702-bb85-4234-84e7-93612e7cf9fe_isrForCrosstalkSources_1125925_62.stderr parsl.dataflow.dflow INFO: Task 1363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e09c5d8f3510c0269ec49f99a92bc2d7 with result from task 1363 parsl.dataflow.dflow DEBUG: Task 1363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/5e5af668-3765-4a46-a51a-85fda1b5d943_isrForCrosstalkSources_1125903_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/5e5af668-3765-4a46-a51a-85fda1b5d943_isrForCrosstalkSources_1125903_41.stderr parsl.dataflow.dflow INFO: Task 1571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e50569a6ce14181650b83963921561d1 with result from task 1571 parsl.dataflow.dflow DEBUG: Task 1571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/fdccedcf-d4d8-4b8f-843a-42c48bbbd085_isrForCrosstalkSources_1125928_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/fdccedcf-d4d8-4b8f-843a-42c48bbbd085_isrForCrosstalkSources_1125928_26.stderr parsl.dataflow.dflow INFO: Task 1576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 267619e936bddbd7ff4ff0e37f6377e1 with result from task 1576 parsl.dataflow.dflow DEBUG: Task 1576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/26fa1884-7bdf-4e95-9eb9-176da691d89c_isrForCrosstalkSources_1125904_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/26fa1884-7bdf-4e95-9eb9-176da691d89c_isrForCrosstalkSources_1125904_26.stderr parsl.dataflow.dflow INFO: Task 1325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c99b4922ad1aba323b95eb68240aa016 with result from task 1325 parsl.dataflow.dflow DEBUG: Task 1325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/2c5e06b6-7dc2-43a9-9afc-92810d476df3_isrForCrosstalkSources_1125889_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/2c5e06b6-7dc2-43a9-9afc-92810d476df3_isrForCrosstalkSources_1125889_40.stderr parsl.dataflow.dflow INFO: Task 1299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91f9fe6f542953bc5bcbeea08340b78b with result from task 1299 parsl.dataflow.dflow DEBUG: Task 1299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/bacf9355-ac97-4d51-a2d6-de9ce4188045_isrForCrosstalkSources_1125943_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/bacf9355-ac97-4d51-a2d6-de9ce4188045_isrForCrosstalkSources_1125943_48.stderr parsl.dataflow.dflow INFO: Task 1583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33d9881c3cbdc38013dac3886aaa95d3 with result from task 1583 parsl.dataflow.dflow DEBUG: Task 1583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/06e5997a-52ce-469b-9adb-97680884b881_isrForCrosstalkSources_1125923_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/06e5997a-52ce-469b-9adb-97680884b881_isrForCrosstalkSources_1125923_11.stderr parsl.dataflow.dflow INFO: Task 1259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f0c107a197ab55466927d773ea234fa with result from task 1259 parsl.dataflow.dflow DEBUG: Task 1259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e169e421-eebf-4e29-ba26-76b5fc18b90c_isrForCrosstalkSources_1125942_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e169e421-eebf-4e29-ba26-76b5fc18b90c_isrForCrosstalkSources_1125942_57.stderr parsl.dataflow.dflow INFO: Task 1374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24900b6c71a330b98ebf6295e3e50cea with result from task 1374 parsl.dataflow.dflow DEBUG: Task 1374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/61205585-ca3c-448b-bfed-5f6ed8ef2727_isrForCrosstalkSources_1125900_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/61205585-ca3c-448b-bfed-5f6ed8ef2727_isrForCrosstalkSources_1125900_55.stderr parsl.dataflow.dflow INFO: Task 1245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a7ac37cf69b0f6cb2f63b9b7a9c0f22 with result from task 1245 parsl.dataflow.dflow DEBUG: Task 1245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/b41d586a-32bc-405c-87e4-e6bafb5da769_isrForCrosstalkSources_1125939_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/b41d586a-32bc-405c-87e4-e6bafb5da769_isrForCrosstalkSources_1125939_52.stderr parsl.dataflow.dflow INFO: Task 1352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9d02da726f96b997f62357c56e02849 with result from task 1352 parsl.dataflow.dflow DEBUG: Task 1352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/e775f15e-5882-4b4b-96a0-013d3528f0e4_isrForCrosstalkSources_1125910_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/e775f15e-5882-4b4b-96a0-013d3528f0e4_isrForCrosstalkSources_1125910_41.stderr parsl.dataflow.dflow INFO: Task 1366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9db4202212cfebad38fff18c625b49c0 with result from task 1366 parsl.dataflow.dflow DEBUG: Task 1366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/4f7fd9d4-f7d1-4f46-946a-4db13e423148_isrForCrosstalkSources_1125947_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/4f7fd9d4-f7d1-4f46-946a-4db13e423148_isrForCrosstalkSources_1125947_43.stderr parsl.dataflow.dflow INFO: Task 1581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae0f182f2ae7dd00c7938e1e0a0d6ade with result from task 1581 parsl.dataflow.dflow DEBUG: Task 1581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/ffce5a6c-6794-4e6a-b6d8-4d1b1fd079c9_isrForCrosstalkSources_1125935_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/ffce5a6c-6794-4e6a-b6d8-4d1b1fd079c9_isrForCrosstalkSources_1125935_58.stderr parsl.dataflow.dflow INFO: Task 1302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5097e00d7f3babd77a7f456759bd7070 with result from task 1302 parsl.dataflow.dflow DEBUG: Task 1302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/aa2855bc-764d-4bbe-a701-9ca87ead0ce5_isrForCrosstalkSources_1125946_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/aa2855bc-764d-4bbe-a701-9ca87ead0ce5_isrForCrosstalkSources_1125946_38.stderr parsl.dataflow.dflow INFO: Task 1391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 869882de80b44cd6afbfe4158566aa3e with result from task 1391 parsl.dataflow.dflow DEBUG: Task 1391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/f2be6914-59dc-4747-b9a8-a7f3a1a55145_isrForCrosstalkSources_1125909_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/f2be6914-59dc-4747-b9a8-a7f3a1a55145_isrForCrosstalkSources_1125909_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:05 proc_lsst.multi:146 [INFO] found job 16896596 in provider checkpoint proc_lsst.multi INFO: found job 16896596 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5f263783b0cf4fe88395d34d299340f with result from task 1527 parsl.dataflow.dflow DEBUG: Task 1527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/fc76317f-ebb9-4c33-9f20-ac7e35e54b05_isrForCrosstalkSources_1125901_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/fc76317f-ebb9-4c33-9f20-ac7e35e54b05_isrForCrosstalkSources_1125901_20.stderr parsl.dataflow.dflow INFO: Task 1390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 084b654e792ae5de6251d463afbbae48 with result from task 1390 parsl.dataflow.dflow DEBUG: Task 1390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a96232ed-23de-4bdb-a4e5-784b5a1dccb7_isrForCrosstalkSources_1125938_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a96232ed-23de-4bdb-a4e5-784b5a1dccb7_isrForCrosstalkSources_1125938_60.stderr parsl.dataflow.dflow INFO: Task 1411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 819f7fac8e7fab6b593692676ac803be with result from task 1411 parsl.dataflow.dflow DEBUG: Task 1411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/ae2cdade-6c68-4e67-b1f4-ecc90472388d_isrForCrosstalkSources_1125909_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/ae2cdade-6c68-4e67-b1f4-ecc90472388d_isrForCrosstalkSources_1125909_16.stderr parsl.dataflow.dflow INFO: Task 1414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 812cac4dd90dee9851d8fa41e1cea0fe with result from task 1414 parsl.dataflow.dflow DEBUG: Task 1414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/b09137f4-6fcb-417c-aed2-92456a191feb_isrForCrosstalkSources_1125923_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/b09137f4-6fcb-417c-aed2-92456a191feb_isrForCrosstalkSources_1125923_5.stderr parsl.dataflow.dflow INFO: Task 1412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33406869dd118f61e4358d55f911cd32 with result from task 1412 parsl.dataflow.dflow DEBUG: Task 1412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/c199e52e-d575-4dd0-be99-d40087b33d41_isrForCrosstalkSources_1125906_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/c199e52e-d575-4dd0-be99-d40087b33d41_isrForCrosstalkSources_1125906_36.stderr parsl.dataflow.dflow INFO: Task 1416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10e45a9459e801b4134cbb15cf81d638 with result from task 1416 parsl.dataflow.dflow DEBUG: Task 1416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/e9f05a7d-2244-49f1-86f0-d2ce1fa49c3a_isrForCrosstalkSources_1125917_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/e9f05a7d-2244-49f1-86f0-d2ce1fa49c3a_isrForCrosstalkSources_1125917_45.stderr parsl.dataflow.dflow INFO: Task 1563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73294469d2cb352842a5d1405a0c00e7 with result from task 1563 parsl.dataflow.dflow DEBUG: Task 1563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/7dd49a0a-1238-4375-bccf-d2caff4f515e_isrForCrosstalkSources_1125907_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/7dd49a0a-1238-4375-bccf-d2caff4f515e_isrForCrosstalkSources_1125907_54.stderr parsl.dataflow.dflow INFO: Task 1395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc648a069617290cca0b13ebe3875a51 with result from task 1395 parsl.dataflow.dflow DEBUG: Task 1395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/cd88cf6a-18cc-4158-bc0c-2d4a95c280d7_isrForCrosstalkSources_1125931_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/cd88cf6a-18cc-4158-bc0c-2d4a95c280d7_isrForCrosstalkSources_1125931_62.stderr parsl.dataflow.dflow INFO: Task 1401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d08fe4e6853655d2fc2dfb6e0ea5b545 with result from task 1401 parsl.dataflow.dflow DEBUG: Task 1401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/6c717a7d-b0e8-4867-b021-d226d4820886_isrForCrosstalkSources_1125914_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/6c717a7d-b0e8-4867-b021-d226d4820886_isrForCrosstalkSources_1125914_8.stderr parsl.dataflow.dflow INFO: Task 1402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32a4f8250b4eed20cfdc4b602d792ea2 with result from task 1402 parsl.dataflow.dflow DEBUG: Task 1402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/b37d4f84-fb8e-4995-9c20-faec38961f06_isrForCrosstalkSources_1125915_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/b37d4f84-fb8e-4995-9c20-faec38961f06_isrForCrosstalkSources_1125915_39.stderr parsl.dataflow.dflow INFO: Task 1400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4fe675b36971caee45403aff40172f6 with result from task 1400 parsl.dataflow.dflow DEBUG: Task 1400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/ba49327c-4e66-4094-a384-375d36f9470a_isrForCrosstalkSources_1125892_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/ba49327c-4e66-4094-a384-375d36f9470a_isrForCrosstalkSources_1125892_15.stderr parsl.dataflow.dflow INFO: Task 1403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 630eaa3a625dde32630332cc3dacfa47 with result from task 1403 parsl.dataflow.dflow DEBUG: Task 1403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/53c6e8ad-2305-4cb1-843d-86031797e63f_isrForCrosstalkSources_1125926_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/53c6e8ad-2305-4cb1-843d-86031797e63f_isrForCrosstalkSources_1125926_9.stderr parsl.dataflow.dflow INFO: Task 1405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1943eaa7a217978632e12a026f3d6aa with result from task 1405 parsl.dataflow.dflow DEBUG: Task 1405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/1590cbce-d53b-4acb-aec4-8f961a2bf582_isrForCrosstalkSources_1125943_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/1590cbce-d53b-4acb-aec4-8f961a2bf582_isrForCrosstalkSources_1125943_13.stderr parsl.dataflow.dflow INFO: Task 1406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66260238ec4c847933da1f2f7a7c2f89 with result from task 1406 parsl.dataflow.dflow DEBUG: Task 1406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/dd05369b-0577-4ba0-90d3-0c15e0106eaa_isrForCrosstalkSources_1125925_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/dd05369b-0577-4ba0-90d3-0c15e0106eaa_isrForCrosstalkSources_1125925_34.stderr parsl.dataflow.dflow INFO: Task 1407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1102f1f7eae7509d64d6972abad25c4a with result from task 1407 parsl.dataflow.dflow DEBUG: Task 1407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/adbc05b6-b340-43fa-b3c6-31eefdbfcbe6_isrForCrosstalkSources_1125945_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/adbc05b6-b340-43fa-b3c6-31eefdbfcbe6_isrForCrosstalkSources_1125945_31.stderr parsl.dataflow.dflow INFO: Task 1518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7eba77399b02c702ec7e344b72cbc09 with result from task 1518 parsl.dataflow.dflow DEBUG: Task 1518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/fc921be8-519c-440d-be2f-22ec539c2c17_isrForCrosstalkSources_1125942_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/fc921be8-519c-440d-be2f-22ec539c2c17_isrForCrosstalkSources_1125942_4.stderr parsl.dataflow.dflow INFO: Task 1410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 676748fd617ab9420879684f6ba13fdd with result from task 1410 parsl.dataflow.dflow DEBUG: Task 1410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d5a24c4e-3227-4743-9b65-44ee305691a2_isrForCrosstalkSources_1125934_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d5a24c4e-3227-4743-9b65-44ee305691a2_isrForCrosstalkSources_1125934_28.stderr parsl.dataflow.dflow INFO: Task 1404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98e8119593637c525c51af7aea810933 with result from task 1404 parsl.dataflow.dflow DEBUG: Task 1404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/17d75be1-5b39-40f4-a28b-2c8f04cf8438_isrForCrosstalkSources_1125940_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/17d75be1-5b39-40f4-a28b-2c8f04cf8438_isrForCrosstalkSources_1125940_15.stderr parsl.dataflow.dflow INFO: Task 1523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ce157c85fea00417f51909edb58374c with result from task 1523 parsl.dataflow.dflow DEBUG: Task 1523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/6dfa0ddf-312c-43a4-934e-67ffc2c24f91_isrForCrosstalkSources_1125899_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/6dfa0ddf-312c-43a4-934e-67ffc2c24f91_isrForCrosstalkSources_1125899_33.stderr parsl.dataflow.dflow INFO: Task 1251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c29accc073e0875e4ebc06baa4613f0 with result from task 1251 parsl.dataflow.dflow DEBUG: Task 1251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/b3bc44fa-c280-4de7-9085-b2e38d575b1e_isrForCrosstalkSources_1125908_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/b3bc44fa-c280-4de7-9085-b2e38d575b1e_isrForCrosstalkSources_1125908_6.stderr parsl.dataflow.dflow INFO: Task 1580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69332bde33fc9adc3b2498bdb536d37f with result from task 1580 parsl.dataflow.dflow DEBUG: Task 1580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/df1bf58a-323e-420a-8932-5484b0c39cd0_isrForCrosstalkSources_1125945_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/df1bf58a-323e-420a-8932-5484b0c39cd0_isrForCrosstalkSources_1125945_13.stderr parsl.dataflow.dflow INFO: Task 1554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 317e71ee379820d88de860c0e3349ae9 with result from task 1554 parsl.dataflow.dflow DEBUG: Task 1554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8182d01b-b18a-44cf-a51c-4dd8d5ece67a_isrForCrosstalkSources_1125936_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8182d01b-b18a-44cf-a51c-4dd8d5ece67a_isrForCrosstalkSources_1125936_43.stderr parsl.dataflow.dflow INFO: Task 1310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86de9b88595b9c421231bb9d5fa8d0ba with result from task 1310 parsl.dataflow.dflow DEBUG: Task 1310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/17e2a5f8-5ecc-42d3-9735-84fa78973479_isrForCrosstalkSources_1125935_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/17e2a5f8-5ecc-42d3-9735-84fa78973479_isrForCrosstalkSources_1125935_2.stderr parsl.dataflow.dflow INFO: Task 1540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bfc5a734c4e0cbacaddcab294359609 with result from task 1540 parsl.dataflow.dflow DEBUG: Task 1540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/3c740822-e5ab-49ca-a892-101b1f1b7e99_isrForCrosstalkSources_1125926_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/3c740822-e5ab-49ca-a892-101b1f1b7e99_isrForCrosstalkSources_1125926_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:05 proc_lsst.multi:146 [INFO] found job 16896597 in provider checkpoint proc_lsst.multi INFO: found job 16896597 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b43e15bbf5cc38af810da15932b9db17 with result from task 1314 parsl.dataflow.dflow DEBUG: Task 1314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/ff21f25c-c5e5-404b-a7f6-163018eb4886_isrForCrosstalkSources_1125921_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/ff21f25c-c5e5-404b-a7f6-163018eb4886_isrForCrosstalkSources_1125921_43.stderr parsl.dataflow.dflow INFO: Task 1317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4461a05d45950af14d7b23521f126979 with result from task 1317 parsl.dataflow.dflow DEBUG: Task 1317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/b899de79-06c0-45b0-bd3d-64dcd8223d7b_isrForCrosstalkSources_1125931_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/b899de79-06c0-45b0-bd3d-64dcd8223d7b_isrForCrosstalkSources_1125931_30.stderr parsl.dataflow.dflow INFO: Task 1417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27e4aadfef8888f1efeece6ede995ff8 with result from task 1417 parsl.dataflow.dflow DEBUG: Task 1417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/768251d2-016a-4fa8-9954-792feb58ee9f_isrForCrosstalkSources_1125904_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/768251d2-016a-4fa8-9954-792feb58ee9f_isrForCrosstalkSources_1125904_48.stderr parsl.dataflow.dflow INFO: Task 1574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b9cec1a7a88b695fe2642b859d77aab with result from task 1574 parsl.dataflow.dflow DEBUG: Task 1574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/0089ef97-5f42-4d62-8162-c25ae7306234_isrForCrosstalkSources_1125888_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/0089ef97-5f42-4d62-8162-c25ae7306234_isrForCrosstalkSources_1125888_3.stderr parsl.dataflow.dflow INFO: Task 1520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d769216d2c89cbb4114ae687c7dbf61 with result from task 1520 parsl.dataflow.dflow DEBUG: Task 1520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/46b75d21-0a64-4f90-b197-984589f8f037_isrForCrosstalkSources_1125906_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/46b75d21-0a64-4f90-b197-984589f8f037_isrForCrosstalkSources_1125906_20.stderr parsl.dataflow.dflow INFO: Task 1524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02220cc91c62daf0907270ff27fa4fe0 with result from task 1524 parsl.dataflow.dflow DEBUG: Task 1524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/9dfb14ae-6d64-4d41-9d60-8940180602a7_isrForCrosstalkSources_1125916_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/9dfb14ae-6d64-4d41-9d60-8940180602a7_isrForCrosstalkSources_1125916_46.stderr parsl.dataflow.dflow INFO: Task 1526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 733d21902956c39ad07c9186bccdb65e with result from task 1526 parsl.dataflow.dflow DEBUG: Task 1526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/68d38b6a-f777-4e42-9ab7-c48665c39ab1_isrForCrosstalkSources_1125930_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/68d38b6a-f777-4e42-9ab7-c48665c39ab1_isrForCrosstalkSources_1125930_51.stderr parsl.dataflow.dflow INFO: Task 1315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0864b12201911a30f7bd16062e409b9e with result from task 1315 parsl.dataflow.dflow DEBUG: Task 1315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/bcf44c23-a51e-46cf-a447-7d2339dbf47c_isrForCrosstalkSources_1125942_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/bcf44c23-a51e-46cf-a447-7d2339dbf47c_isrForCrosstalkSources_1125942_5.stderr parsl.dataflow.dflow INFO: Task 1572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5917c46681daa5db53a79f489c9e59d5 with result from task 1572 parsl.dataflow.dflow DEBUG: Task 1572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/e19325cb-9d67-4099-b681-b8b0d429083e_isrForCrosstalkSources_1125907_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/e19325cb-9d67-4099-b681-b8b0d429083e_isrForCrosstalkSources_1125907_7.stderr parsl.dataflow.dflow INFO: Task 1569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfdea86abfdd7406aa60d0ec7479ee9e with result from task 1569 parsl.dataflow.dflow DEBUG: Task 1569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/136bcd04-7a39-4152-9f35-4e79cb96e2a6_isrForCrosstalkSources_1125939_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/136bcd04-7a39-4152-9f35-4e79cb96e2a6_isrForCrosstalkSources_1125939_35.stderr parsl.dataflow.dflow INFO: Task 1561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d0c1d3188fa12a7d9ae4c947b436494 with result from task 1561 parsl.dataflow.dflow DEBUG: Task 1561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/6d641af9-776e-4403-b740-a985071b0ceb_isrForCrosstalkSources_1125907_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/6d641af9-776e-4403-b740-a985071b0ceb_isrForCrosstalkSources_1125907_53.stderr parsl.dataflow.dflow INFO: Task 1535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5799dc28d06549a6641bad28e99a0edc with result from task 1535 parsl.dataflow.dflow DEBUG: Task 1535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/6f4412db-3730-4730-a676-838759d1b34f_isrForCrosstalkSources_1125925_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/6f4412db-3730-4730-a676-838759d1b34f_isrForCrosstalkSources_1125925_10.stderr parsl.dataflow.dflow INFO: Task 1522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2724420fd4c6cd14370574729bebe61 with result from task 1522 parsl.dataflow.dflow DEBUG: Task 1522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6ce586cf-4e95-4fd5-a438-3996b57a4e07_isrForCrosstalkSources_1125909_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6ce586cf-4e95-4fd5-a438-3996b57a4e07_isrForCrosstalkSources_1125909_1.stderr parsl.dataflow.dflow INFO: Task 1364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe36b12ab89ef769e909ca1ba4c35f66 with result from task 1364 parsl.dataflow.dflow DEBUG: Task 1364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/01d360e9-a48a-464e-9c79-05634e49d91b_isrForCrosstalkSources_1125923_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/01d360e9-a48a-464e-9c79-05634e49d91b_isrForCrosstalkSources_1125923_35.stderr parsl.dataflow.dflow INFO: Task 1307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d15a957ab0b2c047169bfad63a01c836 with result from task 1307 parsl.dataflow.dflow DEBUG: Task 1307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/cf47ce0b-fb02-4d39-987d-00d043798115_isrForCrosstalkSources_1125893_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/cf47ce0b-fb02-4d39-987d-00d043798115_isrForCrosstalkSources_1125893_28.stderr parsl.dataflow.dflow INFO: Task 1250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ea6cd59ecfef1c478f690eddcba616b with result from task 1250 parsl.dataflow.dflow DEBUG: Task 1250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/0dfc4cdf-eec4-4fa5-b2ab-e680dc638d56_isrForCrosstalkSources_1125923_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/0dfc4cdf-eec4-4fa5-b2ab-e680dc638d56_isrForCrosstalkSources_1125923_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:06 proc_lsst.multi:146 [INFO] found job 16896598 in provider checkpoint proc_lsst.multi INFO: found job 16896598 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e48f108a93413bf7b7a286ec3a31630 with result from task 1514 parsl.dataflow.dflow DEBUG: Task 1514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/96bedc43-8a05-405d-b4c9-ca997a5da9ac_isrForCrosstalkSources_1125896_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/96bedc43-8a05-405d-b4c9-ca997a5da9ac_isrForCrosstalkSources_1125896_3.stderr parsl.dataflow.dflow INFO: Task 1296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 018bb832b7e1fd21ccb606d1a4849dad with result from task 1296 parsl.dataflow.dflow DEBUG: Task 1296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/b4bab9df-e167-4cc7-86ed-e5804a29e492_isrForCrosstalkSources_1125916_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/b4bab9df-e167-4cc7-86ed-e5804a29e492_isrForCrosstalkSources_1125916_61.stderr parsl.dataflow.dflow INFO: Task 1362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 613fa2d6617c03f65dff215c1a14b5a5 with result from task 1362 parsl.dataflow.dflow DEBUG: Task 1362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/bf4c0c24-b984-48c2-9874-2456c249e54a_isrForCrosstalkSources_1125944_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/bf4c0c24-b984-48c2-9874-2456c249e54a_isrForCrosstalkSources_1125944_54.stderr parsl.dataflow.dflow INFO: Task 1413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f80f71afd10177e31f3eac50ae818d9c with result from task 1413 parsl.dataflow.dflow DEBUG: Task 1413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/3d5c6889-a934-4b25-a0a3-521fe20552bc_isrForCrosstalkSources_1125913_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/3d5c6889-a934-4b25-a0a3-521fe20552bc_isrForCrosstalkSources_1125913_57.stderr parsl.dataflow.dflow INFO: Task 1548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c02cfddf41a2bde784ff44a924d3f6e2 with result from task 1548 parsl.dataflow.dflow DEBUG: Task 1548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/165f3470-29c6-44e3-8e33-9c53969765eb_isrForCrosstalkSources_1125926_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/165f3470-29c6-44e3-8e33-9c53969765eb_isrForCrosstalkSources_1125926_16.stderr parsl.dataflow.dflow INFO: Task 1542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02276a11a3e7cc55070a14c348af40b9 with result from task 1542 parsl.dataflow.dflow DEBUG: Task 1542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/1d2720db-c471-4fff-a5fb-ff25b72accff_isrForCrosstalkSources_1125888_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/1d2720db-c471-4fff-a5fb-ff25b72accff_isrForCrosstalkSources_1125888_27.stderr parsl.dataflow.dflow INFO: Task 1517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e01d572b5ae3bfea063c41bb4181af2 with result from task 1517 parsl.dataflow.dflow DEBUG: Task 1517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/d4d44e2f-728f-4f23-8d1d-91c2c4a79372_isrForCrosstalkSources_1125916_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/d4d44e2f-728f-4f23-8d1d-91c2c4a79372_isrForCrosstalkSources_1125916_55.stderr parsl.dataflow.dflow INFO: Task 1312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfe6346851c433ffe9e1609d87ba6206 with result from task 1312 parsl.dataflow.dflow DEBUG: Task 1312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/540ed04b-2dbf-437c-b9d4-c10aefc1af24_isrForCrosstalkSources_1125891_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/540ed04b-2dbf-437c-b9d4-c10aefc1af24_isrForCrosstalkSources_1125891_50.stderr parsl.dataflow.dflow INFO: Task 1541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81e21c875471d8ba4e544a91f1884949 with result from task 1541 parsl.dataflow.dflow DEBUG: Task 1541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/bc2b3823-7bc5-428b-a5ea-4870de867876_isrForCrosstalkSources_1125946_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/bc2b3823-7bc5-428b-a5ea-4870de867876_isrForCrosstalkSources_1125946_15.stderr parsl.dataflow.dflow INFO: Task 1415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9571e840150b9af3363a87fa7fe9c2da with result from task 1415 parsl.dataflow.dflow DEBUG: Task 1415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/0254a014-9d47-460d-b4a3-49f183f0bb90_isrForCrosstalkSources_1125937_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/0254a014-9d47-460d-b4a3-49f183f0bb90_isrForCrosstalkSources_1125937_55.stderr parsl.dataflow.dflow INFO: Task 1511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 862389673e37a233c58d242ff7b4e671 with result from task 1511 parsl.dataflow.dflow DEBUG: Task 1511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/bad8542f-8f77-490d-b385-ca29851ea41f_isrForCrosstalkSources_1125933_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/bad8542f-8f77-490d-b385-ca29851ea41f_isrForCrosstalkSources_1125933_37.stderr parsl.dataflow.dflow INFO: Task 1515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebfaf1801ec02376335d8fe545879bff with result from task 1515 parsl.dataflow.dflow DEBUG: Task 1515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d96cf3c4-8536-4e83-84cf-8cac15a6986f_isrForCrosstalkSources_1125919_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d96cf3c4-8536-4e83-84cf-8cac15a6986f_isrForCrosstalkSources_1125919_44.stderr parsl.dataflow.dflow INFO: Task 1323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ff2c4a752e7ff4feffa26ebbe4d65da with result from task 1323 parsl.dataflow.dflow DEBUG: Task 1323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/8fa1c53c-0b47-48db-960d-1d860b276444_isrForCrosstalkSources_1125892_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/8fa1c53c-0b47-48db-960d-1d860b276444_isrForCrosstalkSources_1125892_17.stderr parsl.dataflow.dflow INFO: Task 1545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1699d4f1c31feb62b5f40648fdc881d7 with result from task 1545 parsl.dataflow.dflow DEBUG: Task 1545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/f2df39d4-d78f-4218-889e-1ab121e07c03_isrForCrosstalkSources_1125912_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/f2df39d4-d78f-4218-889e-1ab121e07c03_isrForCrosstalkSources_1125912_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 1248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 738212971817ccb048355c1454fe3a6c with result from task 1248 parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:06 proc_lsst.multi:146 [INFO] found job 16896599 in provider checkpoint proc_lsst.multi INFO: found job 16896599 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/cfd79448-bef9-4173-9526-180af367beb1_isrForCrosstalkSources_1125896_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/cfd79448-bef9-4173-9526-180af367beb1_isrForCrosstalkSources_1125896_22.stderr parsl.dataflow.dflow INFO: Task 1551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 908449b04a4cfdb1075063bcc4fc9019 with result from task 1551 parsl.dataflow.dflow DEBUG: Task 1551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/68c8b0ae-95aa-4af3-894c-d5deb741f55d_isrForCrosstalkSources_1125922_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/68c8b0ae-95aa-4af3-894c-d5deb741f55d_isrForCrosstalkSources_1125922_30.stderr parsl.dataflow.dflow INFO: Task 1255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc77ba1fb8ea0923144d44477d53ddeb with result from task 1255 parsl.dataflow.dflow DEBUG: Task 1255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/59542692-4c93-4250-b2b2-9784f31257f2_isrForCrosstalkSources_1125929_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/59542692-4c93-4250-b2b2-9784f31257f2_isrForCrosstalkSources_1125929_31.stderr parsl.dataflow.dflow INFO: Task 1318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b2c3f8f218a3bf7758fc026b69451be with result from task 1318 parsl.dataflow.dflow DEBUG: Task 1318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/781b9ab1-79d7-47fb-9ee8-13d50d71ab39_isrForCrosstalkSources_1125945_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/781b9ab1-79d7-47fb-9ee8-13d50d71ab39_isrForCrosstalkSources_1125945_44.stderr parsl.dataflow.dflow INFO: Task 1336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86464ea74723313ad20a1bad9c22c411 with result from task 1336 parsl.dataflow.dflow DEBUG: Task 1336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/00d29b24-2f62-4806-b55f-ee7c086e6b25_isrForCrosstalkSources_1125927_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/00d29b24-2f62-4806-b55f-ee7c086e6b25_isrForCrosstalkSources_1125927_28.stderr parsl.dataflow.dflow INFO: Task 1319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10cf0f52ce77d79b20cc7fa4d5badd88 with result from task 1319 parsl.dataflow.dflow DEBUG: Task 1319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/6c029321-1cd4-4506-9663-0c0e6ea002eb_isrForCrosstalkSources_1125897_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/6c029321-1cd4-4506-9663-0c0e6ea002eb_isrForCrosstalkSources_1125897_59.stderr parsl.dataflow.dflow INFO: Task 1562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 889ff74a4c71d5816a92d983edef1bd2 with result from task 1562 parsl.dataflow.dflow DEBUG: Task 1562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/48ca9569-897f-4ec4-849c-38f4d155f6ee_isrForCrosstalkSources_1125919_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/48ca9569-897f-4ec4-849c-38f4d155f6ee_isrForCrosstalkSources_1125919_24.stderr parsl.dataflow.dflow INFO: Task 1253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a28ab0429b0564a460c936264cc29c6 with result from task 1253 parsl.dataflow.dflow DEBUG: Task 1253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/67e1e236-da9a-4946-8c6b-a239bca9803c_isrForCrosstalkSources_1125888_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/67e1e236-da9a-4946-8c6b-a239bca9803c_isrForCrosstalkSources_1125888_61.stderr parsl.dataflow.dflow INFO: Task 1328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 114cfb5c255af6d0a244bd7ab6cfcb8b with result from task 1328 parsl.dataflow.dflow DEBUG: Task 1328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/90a874a1-761d-453a-a161-4ea6d3970154_isrForCrosstalkSources_1125927_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/90a874a1-761d-453a-a161-4ea6d3970154_isrForCrosstalkSources_1125927_52.stderr parsl.dataflow.dflow INFO: Task 1335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e9216fad4d4d47f51f01e80fe4e8365 with result from task 1335 parsl.dataflow.dflow DEBUG: Task 1335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/161413ea-00fc-4b81-867e-65ab0c228f47_isrForCrosstalkSources_1125925_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/161413ea-00fc-4b81-867e-65ab0c228f47_isrForCrosstalkSources_1125925_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:07 proc_lsst.multi:146 [INFO] found job 16896600 in provider checkpoint proc_lsst.multi INFO: found job 16896600 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 813f5ad59f98232990991002f7baf46a with result from task 1324 parsl.dataflow.dflow DEBUG: Task 1324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/88e35c44-4a4c-420d-b5d1-7f5082c75aba_isrForCrosstalkSources_1125903_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/88e35c44-4a4c-420d-b5d1-7f5082c75aba_isrForCrosstalkSources_1125903_9.stderr parsl.dataflow.dflow INFO: Task 1329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 368585fd6209b7eeb3cf680ddda4e4eb with result from task 1329 parsl.dataflow.dflow DEBUG: Task 1329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/1635a302-c05d-42d3-a86d-e163fd9434ae_isrForCrosstalkSources_1125935_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/1635a302-c05d-42d3-a86d-e163fd9434ae_isrForCrosstalkSources_1125935_9.stderr parsl.dataflow.dflow INFO: Task 1539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d90ca67b64cc37fe509a6e890e858bda with result from task 1539 parsl.dataflow.dflow DEBUG: Task 1539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/5ab82810-0fca-4276-860e-6ab43db0b933_isrForCrosstalkSources_1125889_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/5ab82810-0fca-4276-860e-6ab43db0b933_isrForCrosstalkSources_1125889_57.stderr parsl.dataflow.dflow INFO: Task 1584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01d771010e7a9c93677c5e004848620a with result from task 1584 parsl.dataflow.dflow DEBUG: Task 1584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/b71db0f5-6e2a-4f1b-bcb3-c1fa7e9fd455_isrForCrosstalkSources_1125924_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/b71db0f5-6e2a-4f1b-bcb3-c1fa7e9fd455_isrForCrosstalkSources_1125924_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:07 proc_lsst.multi:146 [INFO] found job 16896601 in provider checkpoint proc_lsst.multi INFO: found job 16896601 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6468a64254306ad0bf279cb8210e9f9e with result from task 1333 parsl.dataflow.dflow DEBUG: Task 1333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/0fcf56d1-1d1b-48fb-8a5f-2a91f80a0374_isrForCrosstalkSources_1125938_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/0fcf56d1-1d1b-48fb-8a5f-2a91f80a0374_isrForCrosstalkSources_1125938_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:08 proc_lsst.multi:146 [INFO] found job 16896602 in provider checkpoint proc_lsst.multi INFO: found job 16896602 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f38691e6994c3b0c453cce6d8f8c6209 with result from task 1327 parsl.dataflow.dflow DEBUG: Task 1327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/d9b7f00f-ae5b-437a-bdd3-ec25a187ef3e_isrForCrosstalkSources_1125927_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/d9b7f00f-ae5b-437a-bdd3-ec25a187ef3e_isrForCrosstalkSources_1125927_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:08 proc_lsst.multi:146 [INFO] found job 16896603 in provider checkpoint proc_lsst.multi INFO: found job 16896603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b605145d5a88c1a772d34e357fcb52b6 with result from task 1334 parsl.dataflow.dflow DEBUG: Task 1334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/802b1dee-c621-46b8-9d44-732d2259b570_isrForCrosstalkSources_1125904_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/802b1dee-c621-46b8-9d44-732d2259b570_isrForCrosstalkSources_1125904_29.stderr parsl.dataflow.dflow INFO: Task 1421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d031b159a403c3921b09d0e483c7ae6e with result from task 1421 parsl.dataflow.dflow DEBUG: Task 1421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/8c1cfb76-6b49-4c58-acc8-850593f5e91e_isrForCrosstalkSources_1125890_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/8c1cfb76-6b49-4c58-acc8-850593f5e91e_isrForCrosstalkSources_1125890_18.stderr parsl.dataflow.dflow INFO: Task 1425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6933b16658e1c6869a1e11a0bde7ce82 with result from task 1425 parsl.dataflow.dflow DEBUG: Task 1425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/97798546-2d5f-4020-85bb-5eea4bce8ae1_isrForCrosstalkSources_1125909_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/97798546-2d5f-4020-85bb-5eea4bce8ae1_isrForCrosstalkSources_1125909_58.stderr parsl.dataflow.dflow INFO: Task 1422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b68339ad3b053f207f836210627ba7ee with result from task 1422 parsl.dataflow.dflow DEBUG: Task 1422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/81331e89-bc55-48a5-8e98-7ab01ecec6bb_isrForCrosstalkSources_1125900_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/81331e89-bc55-48a5-8e98-7ab01ecec6bb_isrForCrosstalkSources_1125900_47.stderr parsl.dataflow.dflow INFO: Task 1420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 820bebb2cf67b24f3375bf156b19b9a7 with result from task 1420 parsl.dataflow.dflow DEBUG: Task 1420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/e26c76a1-e4b3-40f2-8213-31999581c438_isrForCrosstalkSources_1125940_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/e26c76a1-e4b3-40f2-8213-31999581c438_isrForCrosstalkSources_1125940_1.stderr parsl.dataflow.dflow INFO: Task 1426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7197a2c60c9e86ed1deb9b1ba4afa39b with result from task 1426 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:08 proc_lsst.multi:146 [INFO] found job 16896604 in provider checkpoint proc_lsst.multi INFO: found job 16896604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/430ae6e6-b786-49a0-9378-1c8933bd3ef5_isrForCrosstalkSources_1125939_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/430ae6e6-b786-49a0-9378-1c8933bd3ef5_isrForCrosstalkSources_1125939_58.stderr parsl.dataflow.dflow INFO: Task 1424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08d0c4fa2d80eeea2362d49cdebb6fa0 with result from task 1424 parsl.dataflow.dflow DEBUG: Task 1424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/935657c2-4341-4944-b62f-511d713da1c0_isrForCrosstalkSources_1125933_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/935657c2-4341-4944-b62f-511d713da1c0_isrForCrosstalkSources_1125933_52.stderr parsl.dataflow.dflow INFO: Task 1419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf881e1e2eb809f624562bed83e8f14c with result from task 1419 parsl.dataflow.dflow DEBUG: Task 1419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/ce9a9547-eaca-41ff-818c-1cccc4719bd4_isrForCrosstalkSources_1125934_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/ce9a9547-eaca-41ff-818c-1cccc4719bd4_isrForCrosstalkSources_1125934_9.stderr parsl.dataflow.dflow INFO: Task 1423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f642ec500c87750fc249970912e09e3 with result from task 1423 parsl.dataflow.dflow DEBUG: Task 1423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/bc21bd57-5589-4807-acb2-30889b1f4b9c_isrForCrosstalkSources_1125924_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/bc21bd57-5589-4807-acb2-30889b1f4b9c_isrForCrosstalkSources_1125924_5.stderr parsl.dataflow.dflow INFO: Task 1587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74b9fbbab09800df8676d583e18d8c56 with result from task 1587 parsl.dataflow.dflow DEBUG: Task 1587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/bd468e3e-f352-43fa-a57d-d10fc2348d65_isrForCrosstalkSources_1125915_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/bd468e3e-f352-43fa-a57d-d10fc2348d65_isrForCrosstalkSources_1125915_46.stderr parsl.dataflow.dflow INFO: Task 1332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a05f087ae8775b91811f84b06f8846f with result from task 1332 parsl.dataflow.dflow DEBUG: Task 1332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/43e2d641-673e-476c-8e65-2261a70e6fde_isrForCrosstalkSources_1125924_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/43e2d641-673e-476c-8e65-2261a70e6fde_isrForCrosstalkSources_1125924_41.stderr parsl.dataflow.dflow INFO: Task 1586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6e7c40620eb093b976ed597194c1a11 with result from task 1586 parsl.dataflow.dflow DEBUG: Task 1586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/3975b170-e48c-403a-9b46-d93783c44b81_isrForCrosstalkSources_1125916_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/3975b170-e48c-403a-9b46-d93783c44b81_isrForCrosstalkSources_1125916_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:08 proc_lsst.multi:146 [INFO] found job 16896605 in provider checkpoint proc_lsst.multi INFO: found job 16896605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c1cc4e5a5d467f4dd908dc1a79a9216 with result from task 1588 parsl.dataflow.dflow DEBUG: Task 1588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/64789d49-7370-4bc2-b302-8ac5bda30f3c_isrForCrosstalkSources_1125924_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/64789d49-7370-4bc2-b302-8ac5bda30f3c_isrForCrosstalkSources_1125924_48.stderr parsl.dataflow.dflow INFO: Task 1330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17ebffce429f1466609ed45c334ed72b with result from task 1330 parsl.dataflow.dflow DEBUG: Task 1330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/86d33459-69a8-4a57-9703-fda100fb9873_isrForCrosstalkSources_1125895_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/86d33459-69a8-4a57-9703-fda100fb9873_isrForCrosstalkSources_1125895_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:09 proc_lsst.multi:146 [INFO] found job 16896606 in provider checkpoint proc_lsst.multi INFO: found job 16896606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a63f060a90dbba8ec003a4cab08cb2f7 with result from task 1337 parsl.dataflow.dflow DEBUG: Task 1337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/b2e050fb-d997-4511-8650-e26f58d5c2a3_isrForCrosstalkSources_1125906_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/b2e050fb-d997-4511-8650-e26f58d5c2a3_isrForCrosstalkSources_1125906_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:09 proc_lsst.multi:146 [INFO] found job 16896607 in provider checkpoint proc_lsst.multi INFO: found job 16896607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a53455a4ae6197182e89901c6dc2295 with result from task 1603 parsl.dataflow.dflow DEBUG: Task 1603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/f557620c-15d2-4289-a207-dc42a45e18f4_isrForCrosstalkSources_1125919_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/f557620c-15d2-4289-a207-dc42a45e18f4_isrForCrosstalkSources_1125919_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:09 proc_lsst.multi:146 [INFO] found job 16896608 in provider checkpoint proc_lsst.multi INFO: found job 16896608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc0c8016e0ef7f4bf4cf204d83447d5f with result from task 1598 parsl.dataflow.dflow DEBUG: Task 1598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/f6182662-3c66-4c3e-90db-0613379b2b47_isrForCrosstalkSources_1125921_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/f6182662-3c66-4c3e-90db-0613379b2b47_isrForCrosstalkSources_1125921_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:09 proc_lsst.multi:146 [INFO] found job 16896609 in provider checkpoint proc_lsst.multi INFO: found job 16896609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:09 proc_lsst.multi:146 [INFO] found job 16896610 in provider checkpoint proc_lsst.multi INFO: found job 16896610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 196f433c5fdc6cb58e4d9e929d9f2727 with result from task 1604 parsl.dataflow.dflow DEBUG: Task 1604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/f5a3028a-a158-42f5-96c7-7fa1cb335596_isrForCrosstalkSources_1125899_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/f5a3028a-a158-42f5-96c7-7fa1cb335596_isrForCrosstalkSources_1125899_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:10 proc_lsst.multi:146 [INFO] found job 16896611 in provider checkpoint proc_lsst.multi INFO: found job 16896611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a036937d8679db63197ca4e056ba0e4b with result from task 1597 parsl.dataflow.dflow DEBUG: Task 1597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/9c360ed8-88c5-428c-9cca-c7ef432118eb_isrForCrosstalkSources_1125932_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/9c360ed8-88c5-428c-9cca-c7ef432118eb_isrForCrosstalkSources_1125932_27.stderr parsl.dataflow.dflow INFO: Task 1685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb2420ef2c5a7b3eec12c94fce0600b1 with result from task 1685 parsl.dataflow.dflow DEBUG: Task 1685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a107e662-a263-408a-b2ad-8001a2ea89c9_isrForCrosstalkSources_1125933_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a107e662-a263-408a-b2ad-8001a2ea89c9_isrForCrosstalkSources_1125933_46.stderr parsl.dataflow.dflow INFO: Task 1624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f411ad29649a90ecf65947eff927de0f with result from task 1624 parsl.dataflow.dflow DEBUG: Task 1624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/1593a835-67b1-4ced-95b0-db5b578dc642_isrForCrosstalkSources_1125938_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/1593a835-67b1-4ced-95b0-db5b578dc642_isrForCrosstalkSources_1125938_19.stderr parsl.dataflow.dflow INFO: Task 1695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8741ec4f56322ae8f74345adae336e2 with result from task 1695 parsl.dataflow.dflow DEBUG: Task 1695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/fc10dd18-73e1-48e0-94e0-232008dea71b_isrForCrosstalkSources_1125941_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/fc10dd18-73e1-48e0-94e0-232008dea71b_isrForCrosstalkSources_1125941_34.stderr parsl.dataflow.dflow INFO: Task 1693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05108ad2651d522d402156d6c2204674 with result from task 1693 parsl.dataflow.dflow DEBUG: Task 1693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/645be1fa-4499-4cb0-a11e-d1c05fd908b8_isrForCrosstalkSources_1125901_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/645be1fa-4499-4cb0-a11e-d1c05fd908b8_isrForCrosstalkSources_1125901_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:10 proc_lsst.multi:146 [INFO] found job 16896612 in provider checkpoint proc_lsst.multi INFO: found job 16896612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 700af268a76b21b4127b4c8bbe911e4b with result from task 1683 parsl.dataflow.dflow DEBUG: Task 1683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/eaf7d378-213e-4b80-8225-5430ad589206_isrForCrosstalkSources_1125913_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/eaf7d378-213e-4b80-8225-5430ad589206_isrForCrosstalkSources_1125913_32.stderr parsl.dataflow.dflow INFO: Task 1642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f8afcf39699eac0f586e049dfcf8753 with result from task 1642 parsl.dataflow.dflow DEBUG: Task 1642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/b02b90b2-ef89-41a4-9052-9c56eb332099_isrForCrosstalkSources_1125901_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/b02b90b2-ef89-41a4-9052-9c56eb332099_isrForCrosstalkSources_1125901_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:10 proc_lsst.multi:146 [INFO] found job 16896613 in provider checkpoint proc_lsst.multi INFO: found job 16896613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09ea8255ff4e0cade9ffbdb839f11b98 with result from task 1688 parsl.dataflow.dflow DEBUG: Task 1688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/3c498bb5-3ca9-497a-be25-abf36a89826b_isrForCrosstalkSources_1125901_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/3c498bb5-3ca9-497a-be25-abf36a89826b_isrForCrosstalkSources_1125901_46.stderr parsl.dataflow.dflow INFO: Task 1702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b83b89a0d154e05de310cae9e72917f with result from task 1702 parsl.dataflow.dflow DEBUG: Task 1702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/7c22d566-5852-439a-96d6-596bf5b7f17b_isrForCrosstalkSources_1125921_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/7c22d566-5852-439a-96d6-596bf5b7f17b_isrForCrosstalkSources_1125921_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:10 proc_lsst.multi:146 [INFO] found job 16896614 in provider checkpoint proc_lsst.multi INFO: found job 16896614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8ad9b66e9620108fcd45ff603b4e575 with result from task 1696 parsl.dataflow.dflow DEBUG: Task 1696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c44e3bcc-229e-4444-a5e6-ef582b430c0e_isrForCrosstalkSources_1125916_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c44e3bcc-229e-4444-a5e6-ef582b430c0e_isrForCrosstalkSources_1125916_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:11 proc_lsst.multi:146 [INFO] found job 16896615 in provider checkpoint proc_lsst.multi INFO: found job 16896615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5820b6bd2204ffed778c56bafbd34b61 with result from task 1697 parsl.dataflow.dflow DEBUG: Task 1697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/80392472-c7a5-43a8-8d78-5de0d1551101_isrForCrosstalkSources_1125913_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/80392472-c7a5-43a8-8d78-5de0d1551101_isrForCrosstalkSources_1125913_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:11 proc_lsst.multi:146 [INFO] found job 16896616 in provider checkpoint proc_lsst.multi INFO: found job 16896616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff0dbdd733fa373ba2ef491814cdabd0 with result from task 1684 parsl.dataflow.dflow DEBUG: Task 1684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/3ea1d29d-44df-4f6b-8fe5-4d802fbb32d6_isrForCrosstalkSources_1125904_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/3ea1d29d-44df-4f6b-8fe5-4d802fbb32d6_isrForCrosstalkSources_1125904_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:11 proc_lsst.multi:146 [INFO] found job 16896617 in provider checkpoint proc_lsst.multi INFO: found job 16896617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe39f56343c5a6bb1add77173c85267a with result from task 1607 parsl.dataflow.dflow DEBUG: Task 1607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/50bf3edf-db6f-44a1-9f7b-c2184fe1c862_isrForCrosstalkSources_1125908_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/50bf3edf-db6f-44a1-9f7b-c2184fe1c862_isrForCrosstalkSources_1125908_54.stderr parsl.dataflow.dflow INFO: Task 1592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d91ed9e6d91ac92bfd3763eac69816fc with result from task 1592 parsl.dataflow.dflow DEBUG: Task 1592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/38aa3d9c-2bb1-49cf-be8a-5823791a7ee7_isrForCrosstalkSources_1125891_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/38aa3d9c-2bb1-49cf-be8a-5823791a7ee7_isrForCrosstalkSources_1125891_27.stderr parsl.dataflow.dflow INFO: Task 1689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcf183011282781e7f62ccbfbb67f919 with result from task 1689 parsl.dataflow.dflow DEBUG: Task 1689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/8d81d651-52f4-4e1d-a4d9-612237b45d3f_isrForCrosstalkSources_1125892_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/8d81d651-52f4-4e1d-a4d9-612237b45d3f_isrForCrosstalkSources_1125892_56.stderr parsl.dataflow.dflow INFO: Task 1680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16458e0e5dc5e5bc3cfe676eb8abf860 with result from task 1680 parsl.dataflow.dflow DEBUG: Task 1680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/c92c4891-9540-4832-9fbe-beed1f548c7a_isrForCrosstalkSources_1125935_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/c92c4891-9540-4832-9fbe-beed1f548c7a_isrForCrosstalkSources_1125935_32.stderr parsl.dataflow.dflow INFO: Task 1643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e8171f82825e2044a6f476ec2c55aec with result from task 1643 parsl.dataflow.dflow DEBUG: Task 1643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/6a78237c-2765-4df4-b775-1497e9621e29_isrForCrosstalkSources_1125932_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/6a78237c-2765-4df4-b775-1497e9621e29_isrForCrosstalkSources_1125932_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:11 proc_lsst.multi:146 [INFO] found job 16896618 in provider checkpoint proc_lsst.multi INFO: found job 16896618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0129acfec9e1dc2593a24e01651f5f06 with result from task 1618 parsl.dataflow.dflow DEBUG: Task 1618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/798f8671-acb8-41cd-b9f2-2f63ec787f16_isrForCrosstalkSources_1125943_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/798f8671-acb8-41cd-b9f2-2f63ec787f16_isrForCrosstalkSources_1125943_14.stderr parsl.dataflow.dflow INFO: Task 1601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0217397fc200c83343520967965cea7b with result from task 1601 parsl.dataflow.dflow DEBUG: Task 1601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/c99f87ed-d265-4f73-8cbb-e48c9aea31ac_isrForCrosstalkSources_1125918_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/c99f87ed-d265-4f73-8cbb-e48c9aea31ac_isrForCrosstalkSources_1125918_57.stderr parsl.dataflow.dflow INFO: Task 1664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 112a2fc5e7bd6e4d537d8f1e0c919f1b with result from task 1664 parsl.dataflow.dflow DEBUG: Task 1664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/4e858019-6cdd-4766-a6a0-7c657fc152c8_isrForCrosstalkSources_1125907_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/4e858019-6cdd-4766-a6a0-7c657fc152c8_isrForCrosstalkSources_1125907_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:12 proc_lsst.multi:146 [INFO] found job 16896619 in provider checkpoint proc_lsst.multi INFO: found job 16896619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0788d91c75e7d8b5c722e73903ccb4a with result from task 1600 parsl.dataflow.dflow DEBUG: Task 1600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/d9a2f24c-41ff-4cf1-9640-af5ba747c5b9_isrForCrosstalkSources_1125908_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/d9a2f24c-41ff-4cf1-9640-af5ba747c5b9_isrForCrosstalkSources_1125908_18.stderr parsl.dataflow.dflow INFO: Task 1608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42d14bc139dc2708e5d36a35c958a1ab with result from task 1608 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 1608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/282f3114-9bd8-4a47-8190-c917da5cdb3e_isrForCrosstalkSources_1125938_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/282f3114-9bd8-4a47-8190-c917da5cdb3e_isrForCrosstalkSources_1125938_34.stderr parsl.dataflow.dflow INFO: Task 1599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 809697c4081435fdec675ee16e25104e with result from task 1599 parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:12 proc_lsst.multi:146 [INFO] found job 16896620 in provider checkpoint proc_lsst.multi INFO: found job 16896620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/47eeb897-15b7-40d5-a0ea-91e117c386ba_isrForCrosstalkSources_1125921_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/47eeb897-15b7-40d5-a0ea-91e117c386ba_isrForCrosstalkSources_1125921_61.stderr parsl.dataflow.dflow INFO: Task 1646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31cc7a4bbbd4b3bb5f6e0cdf0a01602f with result from task 1646 parsl.dataflow.dflow DEBUG: Task 1646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/f9000d28-47a1-42dc-a0d1-2ee1206ff6ca_isrForCrosstalkSources_1125895_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/f9000d28-47a1-42dc-a0d1-2ee1206ff6ca_isrForCrosstalkSources_1125895_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:12 proc_lsst.multi:146 [INFO] found job 16896621 in provider checkpoint proc_lsst.multi INFO: found job 16896621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c5aa98cc1e3e894893f18fbfaeadc05 with result from task 1596 parsl.dataflow.dflow DEBUG: Task 1596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3a519562-726a-4a35-b61a-6a35fdd2fbbc_isrForCrosstalkSources_1125912_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3a519562-726a-4a35-b61a-6a35fdd2fbbc_isrForCrosstalkSources_1125912_17.stderr parsl.dataflow.dflow INFO: Task 1591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5289a1351e34be41d3223cdb6c6635c with result from task 1591 parsl.dataflow.dflow DEBUG: Task 1591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/14ec24fd-3aee-471d-8622-ffcdec1c9481_isrForCrosstalkSources_1125891_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/14ec24fd-3aee-471d-8622-ffcdec1c9481_isrForCrosstalkSources_1125891_39.stderr parsl.dataflow.dflow INFO: Task 1698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 178550ebb6e941433fd3dc6e0a475f6d with result from task 1698 parsl.dataflow.dflow DEBUG: Task 1698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/9affc510-8e67-43fb-b63c-1a27c5a58e62_isrForCrosstalkSources_1125925_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/9affc510-8e67-43fb-b63c-1a27c5a58e62_isrForCrosstalkSources_1125925_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:12 proc_lsst.multi:146 [INFO] found job 16896622 in provider checkpoint proc_lsst.multi INFO: found job 16896622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1aacdd74f3dd9cba6a7dd3c305f8cc62 with result from task 1679 parsl.dataflow.dflow DEBUG: Task 1679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/db7cd409-0205-4c64-8705-5e5e57112ae5_isrForCrosstalkSources_1125934_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/db7cd409-0205-4c64-8705-5e5e57112ae5_isrForCrosstalkSources_1125934_45.stderr parsl.dataflow.dflow INFO: Task 1594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0c34eb408a1d51e0e87fc5e6739fcad with result from task 1594 parsl.dataflow.dflow DEBUG: Task 1594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/f4ca7170-bee1-434b-8e7f-7dc88cc4b2dd_isrForCrosstalkSources_1125914_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/f4ca7170-bee1-434b-8e7f-7dc88cc4b2dd_isrForCrosstalkSources_1125914_55.stderr parsl.dataflow.dflow INFO: Task 1593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d760424a20c76929dcbea0d8382e60dd with result from task 1593 parsl.dataflow.dflow DEBUG: Task 1593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/305bc320-68b9-486a-9a08-b9835130aa15_isrForCrosstalkSources_1125941_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/305bc320-68b9-486a-9a08-b9835130aa15_isrForCrosstalkSources_1125941_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:12 proc_lsst.multi:146 [INFO] found job 16896623 in provider checkpoint proc_lsst.multi INFO: found job 16896623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e80f31b35f7185be850534270130848e with result from task 1648 parsl.dataflow.dflow DEBUG: Task 1648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/6eff3af2-1c6f-4aa1-b780-3e7c7cac58ce_isrForCrosstalkSources_1125910_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/6eff3af2-1c6f-4aa1-b780-3e7c7cac58ce_isrForCrosstalkSources_1125910_25.stderr parsl.dataflow.dflow INFO: Task 1612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a62b5a3c6f43b6851c893bc6ed69d96 with result from task 1612 parsl.dataflow.dflow DEBUG: Task 1612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/38d656f5-ae1f-4e4d-8106-11c938553b1d_isrForCrosstalkSources_1125888_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/38d656f5-ae1f-4e4d-8106-11c938553b1d_isrForCrosstalkSources_1125888_40.stderr parsl.dataflow.dflow INFO: Task 1589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 930d650f6bde1139033b033bf81addc0 with result from task 1589 parsl.dataflow.dflow DEBUG: Task 1589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/6a063722-59dd-44a8-818c-1bda4f44a811_isrForCrosstalkSources_1125889_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/6a063722-59dd-44a8-818c-1bda4f44a811_isrForCrosstalkSources_1125889_16.stderr parsl.dataflow.dflow INFO: Task 1595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23400902371cc7d32b1e1a8e835c71f7 with result from task 1595 parsl.dataflow.dflow DEBUG: Task 1595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0d36ae6d-5f91-4983-9af6-4470c845e157_isrForCrosstalkSources_1125890_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0d36ae6d-5f91-4983-9af6-4470c845e157_isrForCrosstalkSources_1125890_45.stderr parsl.dataflow.dflow INFO: Task 1590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a250e009d82f8bec9f4c86f467540e4b with result from task 1590 parsl.dataflow.dflow DEBUG: Task 1590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/e6b2d57f-5711-4ae6-8f66-8890c95f9604_isrForCrosstalkSources_1125896_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/e6b2d57f-5711-4ae6-8f66-8890c95f9604_isrForCrosstalkSources_1125896_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:13 proc_lsst.multi:146 [INFO] found job 16896624 in provider checkpoint proc_lsst.multi INFO: found job 16896624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2609c978f30e8d5b4e3b33c9b19abd5 with result from task 1609 parsl.dataflow.dflow DEBUG: Task 1609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/1647f9a0-d53e-4e36-8035-1e52b9611bca_isrForCrosstalkSources_1125935_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/1647f9a0-d53e-4e36-8035-1e52b9611bca_isrForCrosstalkSources_1125935_49.stderr parsl.dataflow.dflow INFO: Task 1660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac52ec7215bf6cb1c5700f29aebccfd1 with result from task 1660 parsl.dataflow.dflow DEBUG: Task 1660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/d2b3c902-397e-4987-a382-38695feb219c_isrForCrosstalkSources_1125937_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/d2b3c902-397e-4987-a382-38695feb219c_isrForCrosstalkSources_1125937_39.stderr parsl.dataflow.dflow INFO: Task 1611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb3029d04b96f2b71168e10c7328fd1b with result from task 1611 parsl.dataflow.dflow DEBUG: Task 1611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/06985c0d-d682-487f-a4e2-a235a0e3298d_isrForCrosstalkSources_1125943_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/06985c0d-d682-487f-a4e2-a235a0e3298d_isrForCrosstalkSources_1125943_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:13 proc_lsst.multi:146 [INFO] found job 16896625 in provider checkpoint proc_lsst.multi INFO: found job 16896625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9bde24c921d58226ac941cb0d554706 with result from task 1625 parsl.dataflow.dflow DEBUG: Task 1625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/45ba2658-3f50-44bd-8ef3-3029d89ec7b8_isrForCrosstalkSources_1125922_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/45ba2658-3f50-44bd-8ef3-3029d89ec7b8_isrForCrosstalkSources_1125922_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:13 proc_lsst.multi:146 [INFO] found job 16896626 in provider checkpoint proc_lsst.multi INFO: found job 16896626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4b3eba3d36f4a3244c11dc57cb57736 with result from task 1720 parsl.dataflow.dflow DEBUG: Task 1720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/38c1b795-c023-4c41-a8e9-05fd42e62c7e_isrForCrosstalkSources_1125908_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/38c1b795-c023-4c41-a8e9-05fd42e62c7e_isrForCrosstalkSources_1125908_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:13 proc_lsst.multi:146 [INFO] found job 16896627 in provider checkpoint proc_lsst.multi INFO: found job 16896627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60740e14ad65498d8c253e4c74bd1ec3 with result from task 1655 parsl.dataflow.dflow DEBUG: Task 1655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/7f3a6a80-2711-4146-8a32-e2d985f383ac_isrForCrosstalkSources_1125892_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/7f3a6a80-2711-4146-8a32-e2d985f383ac_isrForCrosstalkSources_1125892_18.stderr parsl.dataflow.dflow INFO: Task 1619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 010d31d5325bf3008ed603e02db5d20a with result from task 1619 parsl.dataflow.dflow DEBUG: Task 1619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/db515f94-ffca-482d-8384-a3fc0d4d4c01_isrForCrosstalkSources_1125944_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/db515f94-ffca-482d-8384-a3fc0d4d4c01_isrForCrosstalkSources_1125944_10.stderr parsl.dataflow.dflow INFO: Task 1694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30ba95904a33e8879b634b1d1e05d252 with result from task 1694 parsl.dataflow.dflow DEBUG: Task 1694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/801333d7-00f2-4ffa-b326-1cf22be50e65_isrForCrosstalkSources_1125917_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/801333d7-00f2-4ffa-b326-1cf22be50e65_isrForCrosstalkSources_1125917_15.stderr parsl.dataflow.dflow INFO: Task 1610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b8f5c3ef4a8e27ad060535851446af3 with result from task 1610 parsl.dataflow.dflow DEBUG: Task 1610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ddad1ffd-ba75-4c1d-8783-79b3db196c64_isrForCrosstalkSources_1125947_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ddad1ffd-ba75-4c1d-8783-79b3db196c64_isrForCrosstalkSources_1125947_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:14 proc_lsst.multi:146 [INFO] found job 16896628 in provider checkpoint proc_lsst.multi INFO: found job 16896628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e720b850a9fe9fbec23569b58c0ec496 with result from task 1644 parsl.dataflow.dflow DEBUG: Task 1644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/10b3e21f-a006-416f-a08f-ddcb8de5a26a_isrForCrosstalkSources_1125889_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/10b3e21f-a006-416f-a08f-ddcb8de5a26a_isrForCrosstalkSources_1125889_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:14 proc_lsst.multi:146 [INFO] found job 16896629 in provider checkpoint proc_lsst.multi INFO: found job 16896629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ffee89700fcd483ff3cb767789d6b0f with result from task 1731 parsl.dataflow.dflow DEBUG: Task 1731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/71e3685e-26fe-4c52-9f35-06921947cbf8_isrForCrosstalkSources_1125894_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/71e3685e-26fe-4c52-9f35-06921947cbf8_isrForCrosstalkSources_1125894_33.stderr parsl.dataflow.dflow INFO: Task 1615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee65f1c1cf7468dfaaf064f2fc50cd22 with result from task 1615 parsl.dataflow.dflow DEBUG: Task 1615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/26f88e4d-f85a-48ba-aca7-1a499e9727bd_isrForCrosstalkSources_1125897_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/26f88e4d-f85a-48ba-aca7-1a499e9727bd_isrForCrosstalkSources_1125897_16.stderr parsl.dataflow.dflow INFO: Task 1628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2413a3a5e1c0282fd4fa1f76c1f42f8f with result from task 1628 parsl.dataflow.dflow DEBUG: Task 1628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/1ca024f6-397c-4128-b614-89c81a16446f_isrForCrosstalkSources_1125926_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/1ca024f6-397c-4128-b614-89c81a16446f_isrForCrosstalkSources_1125926_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:14 proc_lsst.multi:146 [INFO] found job 16896630 in provider checkpoint proc_lsst.multi INFO: found job 16896630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe3e95d3ea75224f590334cbda0b3eca with result from task 1651 parsl.dataflow.dflow DEBUG: Task 1651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/0f987848-8860-4831-9c03-0051e22e4c6a_isrForCrosstalkSources_1125906_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/0f987848-8860-4831-9c03-0051e22e4c6a_isrForCrosstalkSources_1125906_56.stderr parsl.dataflow.dflow INFO: Task 1605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 044462c79ab53ccd5a7b777f831896e7 with result from task 1605 parsl.dataflow.dflow DEBUG: Task 1605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/2eb7a2f4-c031-4c7c-9f89-d4aa4c682990_isrForCrosstalkSources_1125914_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/2eb7a2f4-c031-4c7c-9f89-d4aa4c682990_isrForCrosstalkSources_1125914_53.stderr parsl.dataflow.dflow INFO: Task 1606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2258126d2922c94fb74465bcf1d6f6ad with result from task 1606 parsl.dataflow.dflow DEBUG: Task 1606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2016704d-3ef9-43de-8a97-7f7dac8d2e84_isrForCrosstalkSources_1125933_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2016704d-3ef9-43de-8a97-7f7dac8d2e84_isrForCrosstalkSources_1125933_12.stderr parsl.dataflow.dflow INFO: Task 1645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 410b0b000835e0d4b0467a3d7bedf711 with result from task 1645 parsl.dataflow.dflow DEBUG: Task 1645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/a21bb7ce-f37f-4342-b211-6ebeaccfb3ac_isrForCrosstalkSources_1125920_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/a21bb7ce-f37f-4342-b211-6ebeaccfb3ac_isrForCrosstalkSources_1125920_7.stderr parsl.dataflow.dflow INFO: Task 1741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e600f21305fbc96d979eab53a893191 with result from task 1741 parsl.dataflow.dflow DEBUG: Task 1741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/5ee333d5-207c-4f0b-8508-17599068b64a_isrForCrosstalkSources_1125907_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/5ee333d5-207c-4f0b-8508-17599068b64a_isrForCrosstalkSources_1125907_11.stderr parsl.dataflow.dflow INFO: Task 1706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de9bf32b0e80cd519d4340a288e59e45 with result from task 1706 parsl.dataflow.dflow DEBUG: Task 1706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/b1ad7722-7949-4027-86d3-f7db54ce8d0c_isrForCrosstalkSources_1125922_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/b1ad7722-7949-4027-86d3-f7db54ce8d0c_isrForCrosstalkSources_1125922_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:14 proc_lsst.multi:146 [INFO] found job 16896631 in provider checkpoint proc_lsst.multi INFO: found job 16896631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a6776e39b1035b59f7a91ee74275b88 with result from task 1742 parsl.dataflow.dflow DEBUG: Task 1742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/3c0ae549-df8f-4985-8ad5-e33ca7fda282_isrForCrosstalkSources_1125890_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/3c0ae549-df8f-4985-8ad5-e33ca7fda282_isrForCrosstalkSources_1125890_21.stderr parsl.dataflow.dflow INFO: Task 1766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 331707334b8fe35d0f2bb9b08d6cfcf2 with result from task 1766 parsl.dataflow.dflow DEBUG: Task 1766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/0fdba73d-44a8-4ce6-add9-520bcc2c27b0_isrForCrosstalkSources_1125931_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/0fdba73d-44a8-4ce6-add9-520bcc2c27b0_isrForCrosstalkSources_1125931_13.stderr parsl.dataflow.dflow INFO: Task 1632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 119f084bbebbcc0b7c5585f6ebc586fe with result from task 1632 parsl.dataflow.dflow DEBUG: Task 1632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/5104b4e3-4891-4497-b1a0-8aead38a48ef_isrForCrosstalkSources_1125931_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/5104b4e3-4891-4497-b1a0-8aead38a48ef_isrForCrosstalkSources_1125931_24.stderr parsl.dataflow.dflow INFO: Task 1668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b340b4a801942ec1c23d56809d2139b with result from task 1668 parsl.dataflow.dflow DEBUG: Task 1668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/890ada41-287d-4f4a-8c9e-8844854493a3_isrForCrosstalkSources_1125914_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/890ada41-287d-4f4a-8c9e-8844854493a3_isrForCrosstalkSources_1125914_18.stderr parsl.dataflow.dflow INFO: Task 1662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 164fd3d6aa3e84ae13d944f3576b576f with result from task 1662 parsl.dataflow.dflow DEBUG: Task 1662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/f105db3a-c938-48ad-9c52-d45812a20867_isrForCrosstalkSources_1125911_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/f105db3a-c938-48ad-9c52-d45812a20867_isrForCrosstalkSources_1125911_18.stderr parsl.dataflow.dflow INFO: Task 1708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3db3e72e39693d578a589cfdee431ff with result from task 1708 parsl.dataflow.dflow DEBUG: Task 1708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2e0ee176-9baf-4b01-b889-a5d96737365d_isrForCrosstalkSources_1125933_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2e0ee176-9baf-4b01-b889-a5d96737365d_isrForCrosstalkSources_1125933_36.stderr parsl.dataflow.dflow INFO: Task 1755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1146c76253cd0d0ff356a97dd242f6eb with result from task 1755 parsl.dataflow.dflow DEBUG: Task 1755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/c82e60dd-83a2-4c26-ba4e-54c954884d6e_isrForCrosstalkSources_1125927_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/c82e60dd-83a2-4c26-ba4e-54c954884d6e_isrForCrosstalkSources_1125927_51.stderr parsl.dataflow.dflow INFO: Task 1727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 838f089a0896e422ce08dd0d4cdae8c0 with result from task 1727 parsl.dataflow.dflow DEBUG: Task 1727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/04195519-6b1a-4ea5-a382-a2d8e193f977_isrForCrosstalkSources_1125898_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/04195519-6b1a-4ea5-a382-a2d8e193f977_isrForCrosstalkSources_1125898_1.stderr parsl.dataflow.dflow INFO: Task 1707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d438fafe2b90391306c1cb543c85682c with result from task 1707 parsl.dataflow.dflow DEBUG: Task 1707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/a1892f49-80b3-48e6-bda3-90d0b5be8718_isrForCrosstalkSources_1125912_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/a1892f49-80b3-48e6-bda3-90d0b5be8718_isrForCrosstalkSources_1125912_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:15 proc_lsst.multi:146 [INFO] found job 16896632 in provider checkpoint proc_lsst.multi INFO: found job 16896632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ec6718e2fc959f379b2ead9c53821f5 with result from task 1729 parsl.dataflow.dflow DEBUG: Task 1729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/4a835b6f-7bcd-4437-b10b-89f4cf30ccd7_isrForCrosstalkSources_1125917_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/4a835b6f-7bcd-4437-b10b-89f4cf30ccd7_isrForCrosstalkSources_1125917_19.stderr parsl.dataflow.dflow INFO: Task 1622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa8a9138cc68a7cdd1d20a3594a7f177 with result from task 1622 parsl.dataflow.dflow DEBUG: Task 1622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/793bba36-d31b-45c3-8b2e-f628c79ddf70_isrForCrosstalkSources_1125906_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/793bba36-d31b-45c3-8b2e-f628c79ddf70_isrForCrosstalkSources_1125906_4.stderr parsl.dataflow.dflow INFO: Task 1800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a11c0be247517e8e05df52c0a9df02b0 with result from task 1800 parsl.dataflow.dflow DEBUG: Task 1800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/2eee72d2-2b99-46a2-bf8e-e86fd32099f8_isrForCrosstalkSources_1125935_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/2eee72d2-2b99-46a2-bf8e-e86fd32099f8_isrForCrosstalkSources_1125935_1.stderr parsl.dataflow.dflow INFO: Task 1717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88c44b63e0f0769f2a19612e9acf3a6e with result from task 1717 parsl.dataflow.dflow DEBUG: Task 1717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/b949307b-e86e-4edd-a1c2-978b488d9777_isrForCrosstalkSources_1125946_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/b949307b-e86e-4edd-a1c2-978b488d9777_isrForCrosstalkSources_1125946_26.stderr parsl.dataflow.dflow INFO: Task 1768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b14a281c10829d180293a964f0c03ad with result from task 1768 parsl.dataflow.dflow DEBUG: Task 1768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0f5f0ee4-2cbe-4f5d-9f72-1065fef933fb_isrForCrosstalkSources_1125890_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0f5f0ee4-2cbe-4f5d-9f72-1065fef933fb_isrForCrosstalkSources_1125890_27.stderr parsl.dataflow.dflow INFO: Task 1614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 078e790095ced56e7878be7f23ccfe5c with result from task 1614 parsl.dataflow.dflow DEBUG: Task 1614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/d26eebc4-2a06-44e5-895f-596d0618a63b_isrForCrosstalkSources_1125918_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/d26eebc4-2a06-44e5-895f-596d0618a63b_isrForCrosstalkSources_1125918_10.stderr parsl.dataflow.dflow INFO: Task 1764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b72ea68568b516878d2dec1923304e97 with result from task 1764 parsl.dataflow.dflow DEBUG: Task 1764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/89baa68d-117d-4f69-aa6d-ba25095a7379_isrForCrosstalkSources_1125929_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/89baa68d-117d-4f69-aa6d-ba25095a7379_isrForCrosstalkSources_1125929_33.stderr parsl.dataflow.dflow INFO: Task 1621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d22a36f26c20f52d3b165d8e605ba22 with result from task 1621 parsl.dataflow.dflow DEBUG: Task 1621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/c8b2d7d4-bce3-484a-aef7-0ffe47b38b24_isrForCrosstalkSources_1125935_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/c8b2d7d4-bce3-484a-aef7-0ffe47b38b24_isrForCrosstalkSources_1125935_7.stderr parsl.dataflow.dflow INFO: Task 1661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5304d14325c7ceaf99edd5a3224b5bd7 with result from task 1661 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:15 proc_lsst.multi:146 [INFO] found job 16896633 in provider checkpoint proc_lsst.multi INFO: found job 16896633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/f1c959c4-5e04-4bed-a41b-4b38d472ee64_isrForCrosstalkSources_1125934_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/f1c959c4-5e04-4bed-a41b-4b38d472ee64_isrForCrosstalkSources_1125934_3.stderr parsl.dataflow.dflow INFO: Task 1744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6de1a5da485e51a2904dc0e77a48cec0 with result from task 1744 parsl.dataflow.dflow DEBUG: Task 1744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/123f3939-22fa-4f2b-a481-77cc6f017560_isrForCrosstalkSources_1125945_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/123f3939-22fa-4f2b-a481-77cc6f017560_isrForCrosstalkSources_1125945_7.stderr parsl.dataflow.dflow INFO: Task 1761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98b0a39a4f955322fb1476702a3ec6a4 with result from task 1761 parsl.dataflow.dflow DEBUG: Task 1761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/6cf239e9-c6bd-40a3-a8ae-f825134229a7_isrForCrosstalkSources_1125929_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/6cf239e9-c6bd-40a3-a8ae-f825134229a7_isrForCrosstalkSources_1125929_39.stderr parsl.dataflow.dflow INFO: Task 1779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e24ff427bbf4976945fc7a787f768a5 with result from task 1779 parsl.dataflow.dflow DEBUG: Task 1779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/bd9fc695-1ce2-4c78-8778-3abaccb22b07_isrForCrosstalkSources_1125936_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/bd9fc695-1ce2-4c78-8778-3abaccb22b07_isrForCrosstalkSources_1125936_56.stderr parsl.dataflow.dflow INFO: Task 1629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71bd8619f2c8671ed7aa1bbe166a3e17 with result from task 1629 parsl.dataflow.dflow DEBUG: Task 1629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/6f67ea8e-0dfe-40b6-b4f0-ddc890caa7ec_isrForCrosstalkSources_1125936_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/6f67ea8e-0dfe-40b6-b4f0-ddc890caa7ec_isrForCrosstalkSources_1125936_58.stderr parsl.dataflow.dflow INFO: Task 1616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59ebe2e38b459ac648a6408eb31c013f with result from task 1616 parsl.dataflow.dflow DEBUG: Task 1616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/e1996f19-a4e6-4781-a794-6277d125b156_isrForCrosstalkSources_1125931_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/e1996f19-a4e6-4781-a794-6277d125b156_isrForCrosstalkSources_1125931_6.stderr parsl.dataflow.dflow INFO: Task 1635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9191abaebeebe4693261b43997e64a31 with result from task 1635 parsl.dataflow.dflow DEBUG: Task 1635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/2b171272-0d4c-44bd-aa97-fe5bfb5687cb_isrForCrosstalkSources_1125894_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/2b171272-0d4c-44bd-aa97-fe5bfb5687cb_isrForCrosstalkSources_1125894_4.stderr parsl.dataflow.dflow INFO: Task 1649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47126316c51afbd30ab6a86adc5b5703 with result from task 1649 parsl.dataflow.dflow DEBUG: Task 1649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/5c3c1209-9698-42f2-b4f9-8e0911c1164c_isrForCrosstalkSources_1125943_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/5c3c1209-9698-42f2-b4f9-8e0911c1164c_isrForCrosstalkSources_1125943_10.stderr parsl.dataflow.dflow INFO: Task 1759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b86594dd2819970021ce65f7bf8b48c1 with result from task 1759 parsl.dataflow.dflow DEBUG: Task 1759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/15965065-b664-4ffd-867b-898aa72d4ee8_isrForCrosstalkSources_1125892_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/15965065-b664-4ffd-867b-898aa72d4ee8_isrForCrosstalkSources_1125892_22.stderr parsl.dataflow.dflow INFO: Task 1736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry badc0d30d913c8e0b5400160ea44c7ec with result from task 1736 parsl.dataflow.dflow DEBUG: Task 1736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/126a29ed-c563-4b87-b3b0-cdd27daf9272_isrForCrosstalkSources_1125898_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/126a29ed-c563-4b87-b3b0-cdd27daf9272_isrForCrosstalkSources_1125898_19.stderr parsl.dataflow.dflow INFO: Task 1803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ba6b1777d6438f05621e0f2eacc1fc2 with result from task 1803 parsl.dataflow.dflow DEBUG: Task 1803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/76ba96a1-7271-4d0d-853e-5dac62824692_isrForCrosstalkSources_1125913_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/76ba96a1-7271-4d0d-853e-5dac62824692_isrForCrosstalkSources_1125913_36.stderr parsl.dataflow.dflow INFO: Task 1743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9678ebf0fe8bc29fb8924ffd389f4c91 with result from task 1743 parsl.dataflow.dflow DEBUG: Task 1743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/5b5e4708-5bdf-4477-8ab0-a6186ddc6aeb_isrForCrosstalkSources_1125925_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/5b5e4708-5bdf-4477-8ab0-a6186ddc6aeb_isrForCrosstalkSources_1125925_54.stderr parsl.dataflow.dflow INFO: Task 1617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81e1434cd1d40ef84ff8a7e1c282ffc6 with result from task 1617 parsl.dataflow.dflow DEBUG: Task 1617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/151f9ead-a4b7-4ea4-8e95-751e49edd876_isrForCrosstalkSources_1125895_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/151f9ead-a4b7-4ea4-8e95-751e49edd876_isrForCrosstalkSources_1125895_29.stderr parsl.dataflow.dflow INFO: Task 1730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e5d2023fda39b37fad8d5153bb0f642 with result from task 1730 parsl.dataflow.dflow DEBUG: Task 1730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/8c99813e-85cb-477a-bf1a-a5272998d2ac_isrForCrosstalkSources_1125913_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/8c99813e-85cb-477a-bf1a-a5272998d2ac_isrForCrosstalkSources_1125913_17.stderr parsl.dataflow.dflow INFO: Task 1752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5764ad34f8617ad7e3c46c9dfead0fba with result from task 1752 parsl.dataflow.dflow DEBUG: Task 1752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/baeb05e4-5b05-4364-b1fb-62aad4ac337d_isrForCrosstalkSources_1125917_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/baeb05e4-5b05-4364-b1fb-62aad4ac337d_isrForCrosstalkSources_1125917_9.stderr parsl.dataflow.dflow INFO: Task 1620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c8637e86b03d43296a36d423ec52cdc with result from task 1620 parsl.dataflow.dflow DEBUG: Task 1620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/ea2fb698-ed5a-4cbc-9469-c9f152a7c78b_isrForCrosstalkSources_1125942_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/ea2fb698-ed5a-4cbc-9469-c9f152a7c78b_isrForCrosstalkSources_1125942_50.stderr parsl.dataflow.dflow INFO: Task 1728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6841dcbac5f8089e3334009edd9d92b7 with result from task 1728 parsl.dataflow.dflow DEBUG: Task 1728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/f34d3a51-0dfc-48f7-a0b2-2427eaaa06ab_isrForCrosstalkSources_1125936_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/f34d3a51-0dfc-48f7-a0b2-2427eaaa06ab_isrForCrosstalkSources_1125936_3.stderr parsl.dataflow.dflow INFO: Task 1721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b2dac01ea2cb9fbcf8999826e5a0770 with result from task 1721 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:16 proc_lsst.multi:146 [INFO] found job 16896634 in provider checkpoint proc_lsst.multi INFO: found job 16896634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/e1af1f4c-8773-4972-95e5-a8bd4c9f62cd_isrForCrosstalkSources_1125926_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/e1af1f4c-8773-4972-95e5-a8bd4c9f62cd_isrForCrosstalkSources_1125926_42.stderr parsl.dataflow.dflow INFO: Task 1788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0f5b8cc8d0cedc85ea0741f4828d84b with result from task 1788 parsl.dataflow.dflow DEBUG: Task 1788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/af022f8e-a17c-4134-a62d-fca668882539_isrForCrosstalkSources_1125911_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/af022f8e-a17c-4134-a62d-fca668882539_isrForCrosstalkSources_1125911_8.stderr parsl.dataflow.dflow INFO: Task 1826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c83ce9adf5f597584cc967b6594f58a with result from task 1826 parsl.dataflow.dflow DEBUG: Task 1826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f1e64d90-6e11-44f4-9720-7057e3c2a768_isrForCrosstalkSources_1125894_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f1e64d90-6e11-44f4-9720-7057e3c2a768_isrForCrosstalkSources_1125894_26.stderr parsl.dataflow.dflow INFO: Task 1805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73f79d78d86fda8c0532f8730696a6ff with result from task 1805 parsl.dataflow.dflow DEBUG: Task 1805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/d486716f-58cf-4a9a-85af-7a3e2b02da16_isrForCrosstalkSources_1125902_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/d486716f-58cf-4a9a-85af-7a3e2b02da16_isrForCrosstalkSources_1125902_25.stderr parsl.dataflow.dflow INFO: Task 1754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 794bd96ecd6ae9fa9face1fca1fcf9ae with result from task 1754 parsl.dataflow.dflow DEBUG: Task 1754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/bd63b5fd-6fb3-4b69-987d-987cc1b55358_isrForCrosstalkSources_1125916_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/bd63b5fd-6fb3-4b69-987d-987cc1b55358_isrForCrosstalkSources_1125916_14.stderr parsl.dataflow.dflow INFO: Task 1712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4acfa08f9986840b46f2f264bbc957c with result from task 1712 parsl.dataflow.dflow DEBUG: Task 1712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/e0b75ad0-267f-4d02-bedc-d10a44b12633_isrForCrosstalkSources_1125924_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/e0b75ad0-267f-4d02-bedc-d10a44b12633_isrForCrosstalkSources_1125924_20.stderr parsl.dataflow.dflow INFO: Task 1585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 709ef32f3c7ac6bf35bf4cd3786a532e with result from task 1585 parsl.dataflow.dflow DEBUG: Task 1585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/fb9be19a-0de2-42d2-b4d2-782caec1d68f_isrForCrosstalkSources_1125908_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/fb9be19a-0de2-42d2-b4d2-782caec1d68f_isrForCrosstalkSources_1125908_52.stderr parsl.dataflow.dflow INFO: Task 1816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed011f48c5e0e1e3f9581d1d7605bca5 with result from task 1816 parsl.dataflow.dflow DEBUG: Task 1816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/9c6fb772-7f60-4a08-89f7-fddbfcda9ec4_isrForCrosstalkSources_1125940_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/9c6fb772-7f60-4a08-89f7-fddbfcda9ec4_isrForCrosstalkSources_1125940_25.stderr parsl.dataflow.dflow INFO: Task 1687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf4d08a4bbda9a2ee53e675cb29666f4 with result from task 1687 parsl.dataflow.dflow DEBUG: Task 1687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/27b90af0-56b1-4d82-9542-34f87e1ce231_isrForCrosstalkSources_1125923_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/27b90af0-56b1-4d82-9542-34f87e1ce231_isrForCrosstalkSources_1125923_48.stderr parsl.dataflow.dflow INFO: Task 1815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4085ec0f26848652f06658f044374c8 with result from task 1815 parsl.dataflow.dflow DEBUG: Task 1815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/032d1fe2-b430-4fdc-98ba-1a40c6b7b045_isrForCrosstalkSources_1125944_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/032d1fe2-b430-4fdc-98ba-1a40c6b7b045_isrForCrosstalkSources_1125944_22.stderr parsl.dataflow.dflow INFO: Task 1827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8482e525e9d084bdebe2ae7955467a63 with result from task 1827 parsl.dataflow.dflow DEBUG: Task 1827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/a40598bb-a319-4d50-abf2-ec32b33ae49f_isrForCrosstalkSources_1125904_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/a40598bb-a319-4d50-abf2-ec32b33ae49f_isrForCrosstalkSources_1125904_39.stderr parsl.dataflow.dflow INFO: Task 1781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1875f8b982c0e596b6ac268eb00a16a with result from task 1781 parsl.dataflow.dflow DEBUG: Task 1781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/fd2f334f-114e-496b-8f1d-cae93b1f6362_isrForCrosstalkSources_1125911_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/fd2f334f-114e-496b-8f1d-cae93b1f6362_isrForCrosstalkSources_1125911_56.stderr parsl.dataflow.dflow INFO: Task 1778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f42dd068e1acf76178a4dc3805d3c81 with result from task 1778 parsl.dataflow.dflow DEBUG: Task 1778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/08ba6059-d5b7-4910-af73-04f669ea2673_isrForCrosstalkSources_1125913_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/08ba6059-d5b7-4910-af73-04f669ea2673_isrForCrosstalkSources_1125913_4.stderr parsl.dataflow.dflow INFO: Task 1735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5482820973ab0fbc49f2359aa0e90f72 with result from task 1735 parsl.dataflow.dflow DEBUG: Task 1735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/15e00f63-6670-4b8e-b060-771099601235_isrForCrosstalkSources_1125945_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/15e00f63-6670-4b8e-b060-771099601235_isrForCrosstalkSources_1125945_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:16 proc_lsst.multi:146 [INFO] found job 16896635 in provider checkpoint proc_lsst.multi INFO: found job 16896635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a0dbda7465dab26484f53b43ef0b43f with result from task 1760 parsl.dataflow.dflow DEBUG: Task 1760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/9ed35b3d-d197-4ac7-87dc-0f4b2fe6b876_isrForCrosstalkSources_1125895_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/9ed35b3d-d197-4ac7-87dc-0f4b2fe6b876_isrForCrosstalkSources_1125895_19.stderr parsl.dataflow.dflow INFO: Task 1818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0c58f143995b436039d8e22dc26fd64 with result from task 1818 parsl.dataflow.dflow DEBUG: Task 1818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/35743ccf-88af-4946-abfd-cd5b778a44f5_isrForCrosstalkSources_1125891_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/35743ccf-88af-4946-abfd-cd5b778a44f5_isrForCrosstalkSources_1125891_8.stderr parsl.dataflow.dflow INFO: Task 1797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbf42c025bfe19a0041e594026aa0576 with result from task 1797 parsl.dataflow.dflow DEBUG: Task 1797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/ee8f2feb-e058-45cf-b6d8-16140b6fe05d_isrForCrosstalkSources_1125945_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/ee8f2feb-e058-45cf-b6d8-16140b6fe05d_isrForCrosstalkSources_1125945_47.stderr parsl.dataflow.dflow INFO: Task 1623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 714fe1234900aa8bdb971101198f24b7 with result from task 1623 parsl.dataflow.dflow DEBUG: Task 1623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/3e9746ce-6f27-4c8a-88a8-ee8146576949_isrForCrosstalkSources_1125907_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/3e9746ce-6f27-4c8a-88a8-ee8146576949_isrForCrosstalkSources_1125907_12.stderr parsl.dataflow.dflow INFO: Task 1787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49c04e308e69b62fe9eff69b38d2068d with result from task 1787 parsl.dataflow.dflow DEBUG: Task 1787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c421210f-58bf-444b-b5df-3f969d34cc79_isrForCrosstalkSources_1125904_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c421210f-58bf-444b-b5df-3f969d34cc79_isrForCrosstalkSources_1125904_45.stderr parsl.dataflow.dflow INFO: Task 1613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcc2e02c4bc809206953b0122a7c1218 with result from task 1613 parsl.dataflow.dflow DEBUG: Task 1613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/32e44fb8-aa84-4645-aefb-691397c85b5f_isrForCrosstalkSources_1125901_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/32e44fb8-aa84-4645-aefb-691397c85b5f_isrForCrosstalkSources_1125901_28.stderr parsl.dataflow.dflow INFO: Task 1767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 216ab1df4a221f41c49f222d11c39dc9 with result from task 1767 parsl.dataflow.dflow DEBUG: Task 1767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/35cc4aa9-86e2-4788-8f95-ece4cb20329a_isrForCrosstalkSources_1125916_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/35cc4aa9-86e2-4788-8f95-ece4cb20329a_isrForCrosstalkSources_1125916_29.stderr parsl.dataflow.dflow INFO: Task 1798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3236b40a5b1919d9cbbbbdb6d3015491 with result from task 1798 parsl.dataflow.dflow DEBUG: Task 1798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a4f0d649-be2d-42fc-8d01-da9ea118b70b_isrForCrosstalkSources_1125909_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a4f0d649-be2d-42fc-8d01-da9ea118b70b_isrForCrosstalkSources_1125909_9.stderr parsl.dataflow.dflow INFO: Task 1837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b93876efb66006e5533a03bec8015e8 with result from task 1837 parsl.dataflow.dflow DEBUG: Task 1837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/c895728a-a646-4235-b42b-2476ca9d1d7f_isrForCrosstalkSources_1125931_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/c895728a-a646-4235-b42b-2476ca9d1d7f_isrForCrosstalkSources_1125931_14.stderr parsl.dataflow.dflow INFO: Task 1791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f736999b820faf5f7a20e3fb9298f8e7 with result from task 1791 parsl.dataflow.dflow DEBUG: Task 1791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/a9bb2708-6789-4ff1-a9a1-aa33f74f6f55_isrForCrosstalkSources_1125934_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/a9bb2708-6789-4ff1-a9a1-aa33f74f6f55_isrForCrosstalkSources_1125934_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:17 proc_lsst.multi:146 [INFO] found job 16896636 in provider checkpoint proc_lsst.multi INFO: found job 16896636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d76ce3e4e8f5994893fc6c38205b538e with result from task 1665 parsl.dataflow.dflow DEBUG: Task 1665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/60e643e4-9a5c-46dd-b447-6c524efe370d_isrForCrosstalkSources_1125942_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/60e643e4-9a5c-46dd-b447-6c524efe370d_isrForCrosstalkSources_1125942_41.stderr parsl.dataflow.dflow INFO: Task 1772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 642293ccec56d277667e81f19ca46129 with result from task 1772 parsl.dataflow.dflow DEBUG: Task 1772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/cbfda164-f06f-4215-8220-9e66e949fe11_isrForCrosstalkSources_1125927_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/cbfda164-f06f-4215-8220-9e66e949fe11_isrForCrosstalkSources_1125927_37.stderr parsl.dataflow.dflow INFO: Task 1838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 534bd863ff762c50ffdeda4fb33b1c7c with result from task 1838 parsl.dataflow.dflow DEBUG: Task 1838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/6c68799e-bd19-47f0-a5e0-2d0979d8cfbf_isrForCrosstalkSources_1125936_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/6c68799e-bd19-47f0-a5e0-2d0979d8cfbf_isrForCrosstalkSources_1125936_40.stderr parsl.dataflow.dflow INFO: Task 1602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 954fb0f5bc28b982f7abe39e4d625c5a with result from task 1602 parsl.dataflow.dflow DEBUG: Task 1602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/eeb98cd7-a14a-43ce-903f-917c3cc7cece_isrForCrosstalkSources_1125905_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/eeb98cd7-a14a-43ce-903f-917c3cc7cece_isrForCrosstalkSources_1125905_29.stderr parsl.dataflow.dflow INFO: Task 1821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 288e72258f8d526e55d1080a63ec0a1b with result from task 1821 parsl.dataflow.dflow DEBUG: Task 1821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/afb12a5a-d053-4859-a72b-eadb9fbe0c46_isrForCrosstalkSources_1125889_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/afb12a5a-d053-4859-a72b-eadb9fbe0c46_isrForCrosstalkSources_1125889_10.stderr parsl.dataflow.dflow INFO: Task 1653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 249dae8ffcac75bad25cc89b762e9222 with result from task 1653 parsl.dataflow.dflow DEBUG: Task 1653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/6da3eacd-c970-4c87-a3e1-d539e806e9a9_isrForCrosstalkSources_1125903_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/6da3eacd-c970-4c87-a3e1-d539e806e9a9_isrForCrosstalkSources_1125903_35.stderr parsl.dataflow.dflow INFO: Task 1726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab16eab15b67fea77b299bc9197dce3b with result from task 1726 parsl.dataflow.dflow DEBUG: Task 1726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/32b0581c-6c92-4f5d-88a5-ad4901cbf3ac_isrForCrosstalkSources_1125904_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/32b0581c-6c92-4f5d-88a5-ad4901cbf3ac_isrForCrosstalkSources_1125904_18.stderr parsl.dataflow.dflow INFO: Task 1740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ddb6288ba38aa118486bdb280caddb9 with result from task 1740 parsl.dataflow.dflow DEBUG: Task 1740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/dc3faa2a-0c89-4e51-98d6-53ce2a7d5bfa_isrForCrosstalkSources_1125908_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/dc3faa2a-0c89-4e51-98d6-53ce2a7d5bfa_isrForCrosstalkSources_1125908_4.stderr parsl.dataflow.dflow INFO: Task 1806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cebd163080828a1d733e870ce7eed928 with result from task 1806 parsl.dataflow.dflow DEBUG: Task 1806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e9db2b88-91f8-4b10-a11f-27ac4e8ad660_isrForCrosstalkSources_1125937_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e9db2b88-91f8-4b10-a11f-27ac4e8ad660_isrForCrosstalkSources_1125937_41.stderr parsl.dataflow.dflow INFO: Task 1789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1546536e2d441fc530ca712747b6666 with result from task 1789 parsl.dataflow.dflow DEBUG: Task 1789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/8b380154-b8e2-4d3b-8245-04c45775450b_isrForCrosstalkSources_1125928_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/8b380154-b8e2-4d3b-8245-04c45775450b_isrForCrosstalkSources_1125928_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:18 proc_lsst.multi:146 [INFO] found job 16896637 in provider checkpoint proc_lsst.multi INFO: found job 16896637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c56cf63dff4aef35fa7133b2b9385a9 with result from task 1828 parsl.dataflow.dflow DEBUG: Task 1828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/48550974-1757-4058-b8b3-4567f20b2c79_isrForCrosstalkSources_1125945_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/48550974-1757-4058-b8b3-4567f20b2c79_isrForCrosstalkSources_1125945_38.stderr parsl.dataflow.dflow INFO: Task 1771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4fab33782be59ad758a86642c194433 with result from task 1771 parsl.dataflow.dflow DEBUG: Task 1771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/8527924a-53eb-47bf-a3f2-8fa41510b0d1_isrForCrosstalkSources_1125911_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/8527924a-53eb-47bf-a3f2-8fa41510b0d1_isrForCrosstalkSources_1125911_42.stderr parsl.dataflow.dflow INFO: Task 1804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2fe3b7bba413a022a0bbc75c561d702 with result from task 1804 parsl.dataflow.dflow DEBUG: Task 1804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/cc2f2bcc-9c35-447e-a663-53b043a72af2_isrForCrosstalkSources_1125927_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/cc2f2bcc-9c35-447e-a663-53b043a72af2_isrForCrosstalkSources_1125927_6.stderr parsl.dataflow.dflow INFO: Task 1626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe04a4a1192a4ede6cacc6ab9edb1995 with result from task 1626 parsl.dataflow.dflow DEBUG: Task 1626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/c2c53819-a327-4d38-84ae-9ce0504d23c4_isrForCrosstalkSources_1125907_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/c2c53819-a327-4d38-84ae-9ce0504d23c4_isrForCrosstalkSources_1125907_22.stderr parsl.dataflow.dflow INFO: Task 1819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b41023694f8ceb4a26e12900a5a4c99 with result from task 1819 parsl.dataflow.dflow DEBUG: Task 1819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/650bd64b-b99f-4199-8ccc-143e7506c252_isrForCrosstalkSources_1125942_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/650bd64b-b99f-4199-8ccc-143e7506c252_isrForCrosstalkSources_1125942_1.stderr parsl.dataflow.dflow INFO: Task 1769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3a6b7a154ed09613014f9e1cffdb2b1 with result from task 1769 parsl.dataflow.dflow DEBUG: Task 1769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/671682bf-a5ac-4367-813f-fb10a8c11393_isrForCrosstalkSources_1125934_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/671682bf-a5ac-4367-813f-fb10a8c11393_isrForCrosstalkSources_1125934_24.stderr parsl.dataflow.dflow INFO: Task 1734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1015324698e5ee71d571b7be9f0e64d6 with result from task 1734 parsl.dataflow.dflow DEBUG: Task 1734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/0500b04c-ada3-4eaf-8eb4-9c35f781d930_isrForCrosstalkSources_1125932_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/0500b04c-ada3-4eaf-8eb4-9c35f781d930_isrForCrosstalkSources_1125932_18.stderr parsl.dataflow.dflow INFO: Task 1627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92d19a7f9a69079f1626e5826d0732d8 with result from task 1627 parsl.dataflow.dflow DEBUG: Task 1627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/be55372b-f7c0-4d90-86ee-e990a3036106_isrForCrosstalkSources_1125925_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/be55372b-f7c0-4d90-86ee-e990a3036106_isrForCrosstalkSources_1125925_2.stderr parsl.dataflow.dflow INFO: Task 1808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cab8e390d81728f1e127571adbb2ba3e with result from task 1808 parsl.dataflow.dflow DEBUG: Task 1808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/5c8a5c04-e763-4476-8d16-7ad39d38eb57_isrForCrosstalkSources_1125939_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/5c8a5c04-e763-4476-8d16-7ad39d38eb57_isrForCrosstalkSources_1125939_31.stderr parsl.dataflow.dflow INFO: Task 1634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 468c0fa9f8d07a04b592b97cf3981f2f with result from task 1634 parsl.dataflow.dflow DEBUG: Task 1634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/9312b676-41c9-4c0f-81b0-cbacc53b0d21_isrForCrosstalkSources_1125902_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/9312b676-41c9-4c0f-81b0-cbacc53b0d21_isrForCrosstalkSources_1125902_10.stderr parsl.dataflow.dflow INFO: Task 1785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04be10ab600837650af3ffb38cd5b5c8 with result from task 1785 parsl.dataflow.dflow DEBUG: Task 1785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/18ff9599-06fa-4828-baff-80ad62b1adb8_isrForCrosstalkSources_1125911_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/18ff9599-06fa-4828-baff-80ad62b1adb8_isrForCrosstalkSources_1125911_21.stderr parsl.dataflow.dflow INFO: Task 1796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad2e9283251760f2bbed96f3847d9491 with result from task 1796 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:18 proc_lsst.multi:146 [INFO] found job 16896638 in provider checkpoint proc_lsst.multi INFO: found job 16896638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/f953671d-b506-430c-83de-1dba4f729674_isrForCrosstalkSources_1125899_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/f953671d-b506-430c-83de-1dba4f729674_isrForCrosstalkSources_1125899_9.stderr parsl.dataflow.dflow INFO: Task 1875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f07e965e33829fe81c9e03c3f53d30a with result from task 1875 parsl.dataflow.dflow DEBUG: Task 1875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3b741490-83d9-426a-aa89-356f3a5c9c44_isrForCrosstalkSources_1125920_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3b741490-83d9-426a-aa89-356f3a5c9c44_isrForCrosstalkSources_1125920_30.stderr parsl.dataflow.dflow INFO: Task 1876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 385174f1d8032867b56f4d44b7014158 with result from task 1876 parsl.dataflow.dflow DEBUG: Task 1876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/9f8a10ef-083f-4a64-ab7e-67747cfcdc5b_isrForCrosstalkSources_1125923_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/9f8a10ef-083f-4a64-ab7e-67747cfcdc5b_isrForCrosstalkSources_1125923_39.stderr parsl.dataflow.dflow INFO: Task 1799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af4636e7348de0738c76235692aab650 with result from task 1799 parsl.dataflow.dflow DEBUG: Task 1799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/4080a70f-74d4-4b1d-a16f-c092d6ac561c_isrForCrosstalkSources_1125891_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/4080a70f-74d4-4b1d-a16f-c092d6ac561c_isrForCrosstalkSources_1125891_56.stderr parsl.dataflow.dflow INFO: Task 1691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15c16bbd18f2f36ef2896101b0a5603d with result from task 1691 parsl.dataflow.dflow DEBUG: Task 1691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/bd6b667b-7878-46c4-85c6-abdc14a6b803_isrForCrosstalkSources_1125897_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/bd6b667b-7878-46c4-85c6-abdc14a6b803_isrForCrosstalkSources_1125897_19.stderr parsl.dataflow.dflow INFO: Task 1810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1639c7da6cf9f9db4ec93c03232a345 with result from task 1810 parsl.dataflow.dflow DEBUG: Task 1810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/e397940c-e2ac-4bbe-aeeb-0cef508cd0f0_isrForCrosstalkSources_1125923_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/e397940c-e2ac-4bbe-aeeb-0cef508cd0f0_isrForCrosstalkSources_1125923_60.stderr parsl.dataflow.dflow INFO: Task 1873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cabbe2c05370ad15bf84bfcbef8d1dad with result from task 1873 parsl.dataflow.dflow DEBUG: Task 1873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f19d9da3-94c5-4194-ad4c-7576f8c8a2c0_isrForCrosstalkSources_1125930_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f19d9da3-94c5-4194-ad4c-7576f8c8a2c0_isrForCrosstalkSources_1125930_21.stderr parsl.dataflow.dflow INFO: Task 1639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 534da39f69bbf5b2371a3357dc4ef911 with result from task 1639 parsl.dataflow.dflow DEBUG: Task 1639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/c5bf871b-009b-40af-99db-19e67e816833_isrForCrosstalkSources_1125944_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/c5bf871b-009b-40af-99db-19e67e816833_isrForCrosstalkSources_1125944_2.stderr parsl.dataflow.dflow INFO: Task 1843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e79d5698cb15ffbd2eda38259a77c16c with result from task 1843 parsl.dataflow.dflow DEBUG: Task 1843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/0c9a12eb-4af7-488b-84a3-3a5609b90b0e_isrForCrosstalkSources_1125931_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/0c9a12eb-4af7-488b-84a3-3a5609b90b0e_isrForCrosstalkSources_1125931_51.stderr parsl.dataflow.dflow INFO: Task 1885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97a680f5d8fe5adf1fbbe7447744672f with result from task 1885 parsl.dataflow.dflow DEBUG: Task 1885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/a59d7eba-e17d-408f-91c8-4dbb962ccb1b_isrForCrosstalkSources_1125941_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/a59d7eba-e17d-408f-91c8-4dbb962ccb1b_isrForCrosstalkSources_1125941_19.stderr parsl.dataflow.dflow INFO: Task 1860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3e32b50c04c0174be0c849ee3ebdbf1 with result from task 1860 parsl.dataflow.dflow DEBUG: Task 1860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/6d1f169e-4307-480f-a009-6fadd3ec0eed_isrForCrosstalkSources_1125897_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/6d1f169e-4307-480f-a009-6fadd3ec0eed_isrForCrosstalkSources_1125897_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:18 proc_lsst.multi:146 [INFO] found job 16896639 in provider checkpoint proc_lsst.multi INFO: found job 16896639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 286094600db550c18820328cb33dadf2 with result from task 1807 parsl.dataflow.dflow DEBUG: Task 1807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/4e16b24e-fd53-4d43-9ff3-015cfddc64ef_isrForCrosstalkSources_1125931_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/4e16b24e-fd53-4d43-9ff3-015cfddc64ef_isrForCrosstalkSources_1125931_2.stderr parsl.dataflow.dflow INFO: Task 1750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3665eda571ccb8020cb6b63e3a078641 with result from task 1750 parsl.dataflow.dflow DEBUG: Task 1750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c52f7727-54eb-4639-85c5-8aeaa57f9ccf_isrForCrosstalkSources_1125916_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c52f7727-54eb-4639-85c5-8aeaa57f9ccf_isrForCrosstalkSources_1125916_43.stderr parsl.dataflow.dflow INFO: Task 1692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b03d6ebe0575c6200e011d828a0a7834 with result from task 1692 parsl.dataflow.dflow DEBUG: Task 1692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/d726fb0b-757b-4358-9854-51779374d35d_isrForCrosstalkSources_1125912_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/d726fb0b-757b-4358-9854-51779374d35d_isrForCrosstalkSources_1125912_19.stderr parsl.dataflow.dflow INFO: Task 1880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59a92c41e895131a12362194685b4b9e with result from task 1880 parsl.dataflow.dflow DEBUG: Task 1880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/ce3c9462-4633-4763-b581-337b020fb462_isrForCrosstalkSources_1125931_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/ce3c9462-4633-4763-b581-337b020fb462_isrForCrosstalkSources_1125931_32.stderr parsl.dataflow.dflow INFO: Task 1814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3268b0bff7e4a7b9406017c8d787108 with result from task 1814 parsl.dataflow.dflow DEBUG: Task 1814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/c16b505c-5008-4ac4-8863-f63e7101d8d6_isrForCrosstalkSources_1125900_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/c16b505c-5008-4ac4-8863-f63e7101d8d6_isrForCrosstalkSources_1125900_3.stderr parsl.dataflow.dflow INFO: Task 1780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry decb2c79ef3d78a134ec9a2039b24638 with result from task 1780 parsl.dataflow.dflow DEBUG: Task 1780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/0bc50006-6e75-4763-9c41-af0912d2b3db_isrForCrosstalkSources_1125913_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/0bc50006-6e75-4763-9c41-af0912d2b3db_isrForCrosstalkSources_1125913_25.stderr parsl.dataflow.dflow INFO: Task 1724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47ec1ebe72f683234a98c1a69b19c76c with result from task 1724 parsl.dataflow.dflow DEBUG: Task 1724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/1f5c980e-224c-4b4a-8093-4f516d4ed443_isrForCrosstalkSources_1125911_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/1f5c980e-224c-4b4a-8093-4f516d4ed443_isrForCrosstalkSources_1125911_32.stderr parsl.dataflow.dflow INFO: Task 1640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 320b3691e945f6a89d885a123669dd0a with result from task 1640 parsl.dataflow.dflow DEBUG: Task 1640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/179bd05d-9a53-489d-8015-867f235448f5_isrForCrosstalkSources_1125908_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/179bd05d-9a53-489d-8015-867f235448f5_isrForCrosstalkSources_1125908_47.stderr parsl.dataflow.dflow INFO: Task 1782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3243701e8cf79859aa691c744a9eeba4 with result from task 1782 parsl.dataflow.dflow DEBUG: Task 1782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/04a89edb-f49c-4077-94fb-4e701b984b40_isrForCrosstalkSources_1125940_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/04a89edb-f49c-4077-94fb-4e701b984b40_isrForCrosstalkSources_1125940_61.stderr parsl.dataflow.dflow INFO: Task 1725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69ccfb38aae81092d18c313d0846a36b with result from task 1725 parsl.dataflow.dflow DEBUG: Task 1725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b902671d-0050-40b2-a56d-6520ed0f08dc_isrForCrosstalkSources_1125934_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b902671d-0050-40b2-a56d-6520ed0f08dc_isrForCrosstalkSources_1125934_13.stderr parsl.dataflow.dflow INFO: Task 1811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7ae1a890ba3ca9f72d545c711b993a7 with result from task 1811 parsl.dataflow.dflow DEBUG: Task 1811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/864472bf-81fb-48e5-a951-b6cbf0cd87d7_isrForCrosstalkSources_1125896_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/864472bf-81fb-48e5-a951-b6cbf0cd87d7_isrForCrosstalkSources_1125896_11.stderr parsl.dataflow.dflow INFO: Task 1631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14ebbc4c933908067701b857ea51637e with result from task 1631 parsl.dataflow.dflow DEBUG: Task 1631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ea3afd7e-cbe1-4bba-b2d1-5322e86e8fad_isrForCrosstalkSources_1125939_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ea3afd7e-cbe1-4bba-b2d1-5322e86e8fad_isrForCrosstalkSources_1125939_30.stderr parsl.dataflow.dflow INFO: Task 1839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edec7531f1f60d10bac81b8120b80806 with result from task 1839 parsl.dataflow.dflow DEBUG: Task 1839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/9cac976d-b217-4a10-8ed5-2f165128deb6_isrForCrosstalkSources_1125917_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/9cac976d-b217-4a10-8ed5-2f165128deb6_isrForCrosstalkSources_1125917_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:19 proc_lsst.multi:146 [INFO] found job 16896640 in provider checkpoint proc_lsst.multi INFO: found job 16896640 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e1e11f32fdef7b1320f71e4e785038c with result from task 1802 parsl.dataflow.dflow DEBUG: Task 1802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/bf3d4b58-8851-42a5-bb98-4b7d65da99be_isrForCrosstalkSources_1125903_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/bf3d4b58-8851-42a5-bb98-4b7d65da99be_isrForCrosstalkSources_1125903_50.stderr parsl.dataflow.dflow INFO: Task 1652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c112cfd01ddae0bbb97588708677b243 with result from task 1652 parsl.dataflow.dflow DEBUG: Task 1652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/edf09f1e-f018-40b9-bf3f-0deea3cc87f7_isrForCrosstalkSources_1125932_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/edf09f1e-f018-40b9-bf3f-0deea3cc87f7_isrForCrosstalkSources_1125932_54.stderr parsl.dataflow.dflow INFO: Task 1709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 019737d6b0f8ece9f210fdd9543b87cd with result from task 1709 parsl.dataflow.dflow DEBUG: Task 1709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ce8dcdd7-96fe-4652-ac2a-4208463c9bb2_isrForCrosstalkSources_1125939_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ce8dcdd7-96fe-4652-ac2a-4208463c9bb2_isrForCrosstalkSources_1125939_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:19 proc_lsst.multi:146 [INFO] found job 16896641 in provider checkpoint proc_lsst.multi INFO: found job 16896641 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c95265e92e732679bfa6b2b334acaab with result from task 1656 parsl.dataflow.dflow DEBUG: Task 1656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/5b100163-0290-410e-9e8b-d26595a9a57a_isrForCrosstalkSources_1125918_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/5b100163-0290-410e-9e8b-d26595a9a57a_isrForCrosstalkSources_1125918_32.stderr parsl.dataflow.dflow INFO: Task 1881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d559fdd0b1c7d43aea4ef0bc94619f3a with result from task 1881 parsl.dataflow.dflow DEBUG: Task 1881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/62096c0a-41c3-4fb3-8ee9-c82aac219202_isrForCrosstalkSources_1125921_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/62096c0a-41c3-4fb3-8ee9-c82aac219202_isrForCrosstalkSources_1125921_52.stderr parsl.dataflow.dflow INFO: Task 1719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4724c0c302f4db08ce9274c552b36a0c with result from task 1719 parsl.dataflow.dflow DEBUG: Task 1719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/da63bccc-99d2-4d2c-8073-3ce56dd164fc_isrForCrosstalkSources_1125920_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/da63bccc-99d2-4d2c-8073-3ce56dd164fc_isrForCrosstalkSources_1125920_19.stderr parsl.dataflow.dflow INFO: Task 1745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9e3b4e5e7c69275032d25b0e2f52da4 with result from task 1745 parsl.dataflow.dflow DEBUG: Task 1745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/971417d0-4f63-4eb2-a2bb-4da6ffeaf907_isrForCrosstalkSources_1125927_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/971417d0-4f63-4eb2-a2bb-4da6ffeaf907_isrForCrosstalkSources_1125927_46.stderr parsl.dataflow.dflow INFO: Task 1883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57b5ef994f9e3bd797a7b544ef596a2c with result from task 1883 parsl.dataflow.dflow DEBUG: Task 1883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/54f6e5fa-c490-4f7c-be9c-c0399d6a13c4_isrForCrosstalkSources_1125943_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/54f6e5fa-c490-4f7c-be9c-c0399d6a13c4_isrForCrosstalkSources_1125943_15.stderr parsl.dataflow.dflow INFO: Task 1654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 534a881027528a0926f2f07b1c1d9518 with result from task 1654 parsl.dataflow.dflow DEBUG: Task 1654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0c3d10a3-f6e0-4895-8ea6-fbbfaa598953_isrForCrosstalkSources_1125890_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/0c3d10a3-f6e0-4895-8ea6-fbbfaa598953_isrForCrosstalkSources_1125890_43.stderr parsl.dataflow.dflow INFO: Task 1666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54502e3cf848d9a6fd36c20341a6b396 with result from task 1666 parsl.dataflow.dflow DEBUG: Task 1666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/fd403f33-81f3-487a-9b0e-99ddf329e9fa_isrForCrosstalkSources_1125931_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/fd403f33-81f3-487a-9b0e-99ddf329e9fa_isrForCrosstalkSources_1125931_53.stderr parsl.dataflow.dflow INFO: Task 1674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd1a51fb52a51d3f3d3dca324e2ad9ca with result from task 1674 parsl.dataflow.dflow DEBUG: Task 1674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/870ba7e5-2dce-4ace-bf8c-a917d1d2d794_isrForCrosstalkSources_1125899_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/870ba7e5-2dce-4ace-bf8c-a917d1d2d794_isrForCrosstalkSources_1125899_32.stderr parsl.dataflow.dflow INFO: Task 1786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03e57df19e06367bf5628638d54aabeb with result from task 1786 parsl.dataflow.dflow DEBUG: Task 1786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/eadb659d-b568-47f6-8f44-7f7ff8f88f15_isrForCrosstalkSources_1125913_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/eadb659d-b568-47f6-8f44-7f7ff8f88f15_isrForCrosstalkSources_1125913_43.stderr parsl.dataflow.dflow INFO: Task 1763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 049c05187e4380c14174e02c70c64ec7 with result from task 1763 parsl.dataflow.dflow DEBUG: Task 1763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/666c7029-d787-4958-abd1-7aaf99b7c409_isrForCrosstalkSources_1125939_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/666c7029-d787-4958-abd1-7aaf99b7c409_isrForCrosstalkSources_1125939_59.stderr parsl.dataflow.dflow INFO: Task 1842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb802d7fe9efbc14aacf02a460f755b6 with result from task 1842 parsl.dataflow.dflow DEBUG: Task 1842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/3602efc7-a284-498a-879e-1debf9012305_isrForCrosstalkSources_1125946_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/3602efc7-a284-498a-879e-1debf9012305_isrForCrosstalkSources_1125946_16.stderr parsl.dataflow.dflow INFO: Task 1795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f503fe8a7367c818a8d32ca089cfc222 with result from task 1795 parsl.dataflow.dflow DEBUG: Task 1795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/5d8d597c-2930-44d7-93bf-1c49531d683f_isrForCrosstalkSources_1125900_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/5d8d597c-2930-44d7-93bf-1c49531d683f_isrForCrosstalkSources_1125900_51.stderr parsl.dataflow.dflow INFO: Task 1820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22c0a7042fb8442ce5b59f94bafb7e15 with result from task 1820 parsl.dataflow.dflow DEBUG: Task 1820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/694b6063-319f-4ca3-b340-95a2a3734a06_isrForCrosstalkSources_1125942_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/694b6063-319f-4ca3-b340-95a2a3734a06_isrForCrosstalkSources_1125942_60.stderr parsl.dataflow.dflow INFO: Task 1723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50db344db3f0a892232442fef19853f6 with result from task 1723 parsl.dataflow.dflow DEBUG: Task 1723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/af2da7dd-692d-47b7-8cf6-b1787dacd0b3_isrForCrosstalkSources_1125936_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/af2da7dd-692d-47b7-8cf6-b1787dacd0b3_isrForCrosstalkSources_1125936_45.stderr parsl.dataflow.dflow INFO: Task 1710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25447cd134b0f6365e9d3efc229c4c3c with result from task 1710 parsl.dataflow.dflow DEBUG: Task 1710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/b4d13190-f35c-431a-9679-46b1ed1b557d_isrForCrosstalkSources_1125923_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/b4d13190-f35c-431a-9679-46b1ed1b557d_isrForCrosstalkSources_1125923_28.stderr parsl.dataflow.dflow INFO: Task 1699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ece15dfa40b7e7537a8da1a88c4d781e with result from task 1699 parsl.dataflow.dflow DEBUG: Task 1699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/35d2dfc1-007e-4a53-9c81-084a9a081bbc_isrForCrosstalkSources_1125909_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/35d2dfc1-007e-4a53-9c81-084a9a081bbc_isrForCrosstalkSources_1125909_37.stderr parsl.dataflow.dflow INFO: Task 1849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7de5a50724a867f3bf6e2b86b4dba62 with result from task 1849 parsl.dataflow.dflow DEBUG: Task 1849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/39dc5fa9-98d8-47cb-8624-aece02a25108_isrForCrosstalkSources_1125938_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/39dc5fa9-98d8-47cb-8624-aece02a25108_isrForCrosstalkSources_1125938_38.stderr parsl.dataflow.dflow INFO: Task 1737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92f8913060047c1ec9b7fc16fbe1f409 with result from task 1737 parsl.dataflow.dflow DEBUG: Task 1737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/d8839d1e-fa67-41d7-b631-dbb06422ffc3_isrForCrosstalkSources_1125899_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/d8839d1e-fa67-41d7-b631-dbb06422ffc3_isrForCrosstalkSources_1125899_35.stderr parsl.dataflow.dflow INFO: Task 1890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04514ec5dfaf93d880398826860f4bee with result from task 1890 parsl.dataflow.dflow DEBUG: Task 1890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/8910b4c3-2f57-4fdd-a767-ee2d31e810bd_isrForCrosstalkSources_1125909_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/8910b4c3-2f57-4fdd-a767-ee2d31e810bd_isrForCrosstalkSources_1125909_49.stderr parsl.dataflow.dflow INFO: Task 1854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40a395944c73d4c70c83677f134e112e with result from task 1854 parsl.dataflow.dflow DEBUG: Task 1854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/431e2cd7-b183-486f-87ca-263e953e8bed_isrForCrosstalkSources_1125910_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/431e2cd7-b183-486f-87ca-263e953e8bed_isrForCrosstalkSources_1125910_47.stderr parsl.dataflow.dflow INFO: Task 1853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7846e43d187ab559108d66984c005b13 with result from task 1853 parsl.dataflow.dflow DEBUG: Task 1853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/53c4768b-beb6-44a0-ab67-e31e7f4a0a5d_isrForCrosstalkSources_1125946_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/53c4768b-beb6-44a0-ab67-e31e7f4a0a5d_isrForCrosstalkSources_1125946_54.stderr parsl.dataflow.dflow INFO: Task 1718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17690b9d3379257d4def4e099f8698ac with result from task 1718 parsl.dataflow.dflow DEBUG: Task 1718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/5b25cabe-0e4c-484a-9e16-387af2f72c29_isrForCrosstalkSources_1125933_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/5b25cabe-0e4c-484a-9e16-387af2f72c29_isrForCrosstalkSources_1125933_5.stderr parsl.dataflow.dflow INFO: Task 1879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56a38ad1e3710725cfc8963a706df44a with result from task 1879 parsl.dataflow.dflow DEBUG: Task 1879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/752ddc0a-3bec-472f-9263-9e2dffe6d617_isrForCrosstalkSources_1125943_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/752ddc0a-3bec-472f-9263-9e2dffe6d617_isrForCrosstalkSources_1125943_55.stderr parsl.dataflow.dflow INFO: Task 1847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f4c543a19d8e202430e37e1429d9fe5 with result from task 1847 parsl.dataflow.dflow DEBUG: Task 1847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/0e1507ce-3b96-48df-b992-22366a856870_isrForCrosstalkSources_1125926_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/0e1507ce-3b96-48df-b992-22366a856870_isrForCrosstalkSources_1125926_47.stderr parsl.dataflow.dflow INFO: Task 1667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb26a1c641c766fb98e848af8557b3f6 with result from task 1667 parsl.dataflow.dflow DEBUG: Task 1667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c74d0ff8-fbd0-4684-8d5e-f73c75a339c2_isrForCrosstalkSources_1125904_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c74d0ff8-fbd0-4684-8d5e-f73c75a339c2_isrForCrosstalkSources_1125904_59.stderr parsl.dataflow.dflow INFO: Task 1888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1e1b1baab1a9caddc6adc6341f1e38c with result from task 1888 parsl.dataflow.dflow DEBUG: Task 1888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/ed877f50-5909-49b7-99ac-b4e83b2821d1_isrForCrosstalkSources_1125891_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/ed877f50-5909-49b7-99ac-b4e83b2821d1_isrForCrosstalkSources_1125891_5.stderr parsl.dataflow.dflow INFO: Task 1825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff35ea2560beaddf403b48b6e01eb020 with result from task 1825 parsl.dataflow.dflow DEBUG: Task 1825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/84511bfb-c522-4512-942d-6190973e8693_isrForCrosstalkSources_1125896_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/84511bfb-c522-4512-942d-6190973e8693_isrForCrosstalkSources_1125896_47.stderr parsl.dataflow.dflow INFO: Task 1636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9e05d12d327d91bb8f0808d70fba401 with result from task 1636 parsl.dataflow.dflow DEBUG: Task 1636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a36181da-4186-4288-9c17-6c73509b9956_isrForCrosstalkSources_1125921_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a36181da-4186-4288-9c17-6c73509b9956_isrForCrosstalkSources_1125921_37.stderr parsl.dataflow.dflow INFO: Task 1817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aecf30e5d750fc2ba765a4480c6da825 with result from task 1817 parsl.dataflow.dflow DEBUG: Task 1817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/8a856cf7-be81-4c2e-81bb-7e7665e78d9a_isrForCrosstalkSources_1125937_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/8a856cf7-be81-4c2e-81bb-7e7665e78d9a_isrForCrosstalkSources_1125937_44.stderr parsl.dataflow.dflow INFO: Task 1663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4379314cb9da33497caea4a6a195a73 with result from task 1663 parsl.dataflow.dflow DEBUG: Task 1663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/1d100df8-d656-46e6-b67a-1d68437136d5_isrForCrosstalkSources_1125910_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/1d100df8-d656-46e6-b67a-1d68437136d5_isrForCrosstalkSources_1125910_53.stderr parsl.dataflow.dflow INFO: Task 1638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 709307ead8a4821cc6273d6fd3efa392 with result from task 1638 parsl.dataflow.dflow DEBUG: Task 1638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/b955eaa0-9064-40a2-86ca-799af1182890_isrForCrosstalkSources_1125947_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/b955eaa0-9064-40a2-86ca-799af1182890_isrForCrosstalkSources_1125947_54.stderr parsl.dataflow.dflow INFO: Task 1765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 418a079650114100634eeac93bfbcebb with result from task 1765 parsl.dataflow.dflow DEBUG: Task 1765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/1e0af3b9-3508-4945-a6be-d6861f46a4b3_isrForCrosstalkSources_1125940_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/1e0af3b9-3508-4945-a6be-d6861f46a4b3_isrForCrosstalkSources_1125940_9.stderr parsl.dataflow.dflow INFO: Task 1892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d648a879ff8a02d6541d1c1cf31fcc8e with result from task 1892 parsl.dataflow.dflow DEBUG: Task 1892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b52956ec-2106-4fb2-bb37-480996f5320b_isrForCrosstalkSources_1125938_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b52956ec-2106-4fb2-bb37-480996f5320b_isrForCrosstalkSources_1125938_39.stderr parsl.dataflow.dflow INFO: Task 1637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02910caf7b0437d0496bf149f0e0d0ee with result from task 1637 parsl.dataflow.dflow DEBUG: Task 1637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/61c48643-5150-4c2d-b00e-92a80a602c40_isrForCrosstalkSources_1125945_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/61c48643-5150-4c2d-b00e-92a80a602c40_isrForCrosstalkSources_1125945_3.stderr parsl.dataflow.dflow INFO: Task 1700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88eca61145bc26fd37af58975092913c with result from task 1700 parsl.dataflow.dflow DEBUG: Task 1700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/07300104-0dc1-4577-bcfe-91ccaeb482a7_isrForCrosstalkSources_1125939_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/07300104-0dc1-4577-bcfe-91ccaeb482a7_isrForCrosstalkSources_1125939_16.stderr parsl.dataflow.dflow INFO: Task 1722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 842e5efd1e4fcfbb163c7df07add23bf with result from task 1722 parsl.dataflow.dflow DEBUG: Task 1722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/4ee0f4b9-47b8-4831-b167-450fccb4ecfa_isrForCrosstalkSources_1125902_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/4ee0f4b9-47b8-4831-b167-450fccb4ecfa_isrForCrosstalkSources_1125902_35.stderr parsl.dataflow.dflow INFO: Task 1793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce2780a9e8bf8bdb2242673b204a8e2d with result from task 1793 parsl.dataflow.dflow DEBUG: Task 1793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/baaeeb97-fe53-4613-a71c-c0a40fa6fe46_isrForCrosstalkSources_1125934_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/baaeeb97-fe53-4613-a71c-c0a40fa6fe46_isrForCrosstalkSources_1125934_47.stderr parsl.dataflow.dflow INFO: Task 1758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab92d7f74571ae6f373bbe1686568ae6 with result from task 1758 parsl.dataflow.dflow DEBUG: Task 1758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/99e9f611-b8a4-464b-bcf5-8b5c5854fc55_isrForCrosstalkSources_1125929_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/99e9f611-b8a4-464b-bcf5-8b5c5854fc55_isrForCrosstalkSources_1125929_44.stderr parsl.dataflow.dflow INFO: Task 1801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbd81d0bf8b14c72e91539551ab01071 with result from task 1801 parsl.dataflow.dflow DEBUG: Task 1801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/9947b9d8-572a-4e53-9c61-37b2ffc53c58_isrForCrosstalkSources_1125943_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/9947b9d8-572a-4e53-9c61-37b2ffc53c58_isrForCrosstalkSources_1125943_16.stderr parsl.dataflow.dflow INFO: Task 1749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2dd25750280f8ba1838f25822e0a7bc with result from task 1749 parsl.dataflow.dflow DEBUG: Task 1749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/4635d720-703c-4df4-91f7-fbb697e89f16_isrForCrosstalkSources_1125918_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/4635d720-703c-4df4-91f7-fbb697e89f16_isrForCrosstalkSources_1125918_36.stderr parsl.dataflow.dflow INFO: Task 1851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a11dd403200d1d9397985fe29498b0c8 with result from task 1851 parsl.dataflow.dflow DEBUG: Task 1851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/ae49fb2d-8ca5-43d1-ac28-ff44150e50bd_isrForCrosstalkSources_1125895_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/ae49fb2d-8ca5-43d1-ac28-ff44150e50bd_isrForCrosstalkSources_1125895_38.stderr parsl.dataflow.dflow INFO: Task 1705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6447c83483b84b58578a0b2c86216c68 with result from task 1705 parsl.dataflow.dflow DEBUG: Task 1705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/cde5c015-8742-473c-af09-9915d99c345c_isrForCrosstalkSources_1125921_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/cde5c015-8742-473c-af09-9915d99c345c_isrForCrosstalkSources_1125921_23.stderr parsl.dataflow.dflow INFO: Task 1738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddccdca0aac6c8e2e9f12710be4ea97d with result from task 1738 parsl.dataflow.dflow DEBUG: Task 1738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/fefed67a-a1dd-4614-a7f4-a0c3e41b1dd5_isrForCrosstalkSources_1125932_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/fefed67a-a1dd-4614-a7f4-a0c3e41b1dd5_isrForCrosstalkSources_1125932_55.stderr parsl.dataflow.dflow INFO: Task 1751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23da9fe904bb68d4fddea2da35f3a3b2 with result from task 1751 parsl.dataflow.dflow DEBUG: Task 1751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/35d9ac3c-237d-4d54-b291-9249ad11241c_isrForCrosstalkSources_1125927_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/35d9ac3c-237d-4d54-b291-9249ad11241c_isrForCrosstalkSources_1125927_17.stderr parsl.dataflow.dflow INFO: Task 1704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d115b2e45de842a9571bd7455335cc0 with result from task 1704 parsl.dataflow.dflow DEBUG: Task 1704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/f20e7325-cf81-45de-9657-d2a8537b9f40_isrForCrosstalkSources_1125928_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/f20e7325-cf81-45de-9657-d2a8537b9f40_isrForCrosstalkSources_1125928_21.stderr parsl.dataflow.dflow INFO: Task 1690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea91884e29dea69b5a083938d76b857f with result from task 1690 parsl.dataflow.dflow DEBUG: Task 1690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/a5c9e032-9c5a-450e-86d9-03a4bebf45b3_isrForCrosstalkSources_1125924_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/a5c9e032-9c5a-450e-86d9-03a4bebf45b3_isrForCrosstalkSources_1125924_62.stderr parsl.dataflow.dflow INFO: Task 1861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1be3f20431f5fa26102acd3932828f79 with result from task 1861 parsl.dataflow.dflow DEBUG: Task 1861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4dcc04d6-0202-40c4-9f95-e2e87e93294e_isrForCrosstalkSources_1125908_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4dcc04d6-0202-40c4-9f95-e2e87e93294e_isrForCrosstalkSources_1125908_24.stderr parsl.dataflow.dflow INFO: Task 1836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e87318e09acb09869f6206495c3c9e1e with result from task 1836 parsl.dataflow.dflow DEBUG: Task 1836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/670abf8e-e874-49ba-aa29-ac2bbbebc16b_isrForCrosstalkSources_1125920_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/670abf8e-e874-49ba-aa29-ac2bbbebc16b_isrForCrosstalkSources_1125920_32.stderr parsl.dataflow.dflow INFO: Task 1658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74d91ed240359c100d19e714869cf9a1 with result from task 1658 parsl.dataflow.dflow DEBUG: Task 1658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/55476aba-9ef8-4746-9bd0-4e1181ba53c0_isrForCrosstalkSources_1125893_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/55476aba-9ef8-4746-9bd0-4e1181ba53c0_isrForCrosstalkSources_1125893_46.stderr parsl.dataflow.dflow INFO: Task 1872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dc734363f9a3eaf5e3d262e1ddd9c2c with result from task 1872 parsl.dataflow.dflow DEBUG: Task 1872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f6e486d5-6750-4b54-95e9-d3ab08fcd4bf_isrForCrosstalkSources_1125897_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f6e486d5-6750-4b54-95e9-d3ab08fcd4bf_isrForCrosstalkSources_1125897_33.stderr parsl.dataflow.dflow INFO: Task 1703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37c16425d7df34b42009f34cbb8e4b8a with result from task 1703 parsl.dataflow.dflow DEBUG: Task 1703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/aa1d4849-95e3-407a-a9c9-e95d63a12a6f_isrForCrosstalkSources_1125940_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/aa1d4849-95e3-407a-a9c9-e95d63a12a6f_isrForCrosstalkSources_1125940_17.stderr parsl.dataflow.dflow INFO: Task 1841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44fa9b26427f4f37977699120ff4e0ff with result from task 1841 parsl.dataflow.dflow DEBUG: Task 1841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/156cc795-55fd-438d-aec4-7a3d5ac872f1_isrForCrosstalkSources_1125908_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/156cc795-55fd-438d-aec4-7a3d5ac872f1_isrForCrosstalkSources_1125908_38.stderr parsl.dataflow.dflow INFO: Task 1701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 782c805f18fb5e181e7438538472304b with result from task 1701 parsl.dataflow.dflow DEBUG: Task 1701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/2bbbb378-3585-4e30-9c81-94bf418c3cdc_isrForCrosstalkSources_1125897_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/2bbbb378-3585-4e30-9c81-94bf418c3cdc_isrForCrosstalkSources_1125897_7.stderr parsl.dataflow.dflow INFO: Task 1840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a69906461e191df6ecfd22682b0eb3b2 with result from task 1840 parsl.dataflow.dflow DEBUG: Task 1840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/fa7cc0d2-05d2-40b5-b787-21ea29f62977_isrForCrosstalkSources_1125944_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/fa7cc0d2-05d2-40b5-b787-21ea29f62977_isrForCrosstalkSources_1125944_5.stderr parsl.dataflow.dflow INFO: Task 1659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe2bf648daf3b2489d2bf46b25cf1369 with result from task 1659 parsl.dataflow.dflow DEBUG: Task 1659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/7c57323f-1266-4647-9474-0574bca97bb2_isrForCrosstalkSources_1125911_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/7c57323f-1266-4647-9474-0574bca97bb2_isrForCrosstalkSources_1125911_46.stderr parsl.dataflow.dflow INFO: Task 1871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03aba18fa49c8e58e3d8d9fa39c3c7d0 with result from task 1871 parsl.dataflow.dflow DEBUG: Task 1871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/ad84d27d-5e9b-42a5-9478-6985840f1a46_isrForCrosstalkSources_1125899_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/ad84d27d-5e9b-42a5-9478-6985840f1a46_isrForCrosstalkSources_1125899_19.stderr parsl.dataflow.dflow INFO: Task 1630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8ce89ff5586f2a05c777342fa9d82e6 with result from task 1630 parsl.dataflow.dflow DEBUG: Task 1630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/f9394f79-b06b-4b55-ab64-84f7d6de52d3_isrForCrosstalkSources_1125929_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/f9394f79-b06b-4b55-ab64-84f7d6de52d3_isrForCrosstalkSources_1125929_58.stderr parsl.dataflow.dflow INFO: Task 1671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc72aaddcbdc560873bf340483baa945 with result from task 1671 parsl.dataflow.dflow DEBUG: Task 1671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/b4ab17da-d738-4c2a-88c0-c210b291eca0_isrForCrosstalkSources_1125925_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/b4ab17da-d738-4c2a-88c0-c210b291eca0_isrForCrosstalkSources_1125925_11.stderr parsl.dataflow.dflow INFO: Task 1647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5535a5d9828ae8704c11da0c65a85d0 with result from task 1647 parsl.dataflow.dflow DEBUG: Task 1647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8cc53f6c-27e9-44d7-8d66-8a4a6257a716_isrForCrosstalkSources_1125891_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8cc53f6c-27e9-44d7-8d66-8a4a6257a716_isrForCrosstalkSources_1125891_16.stderr parsl.dataflow.dflow INFO: Task 1866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1235dd019750e0c1320e3f4a63bcddb with result from task 1866 parsl.dataflow.dflow DEBUG: Task 1866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/3c4eaa79-986b-453b-a4bb-338a267965a7_isrForCrosstalkSources_1125895_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/3c4eaa79-986b-453b-a4bb-338a267965a7_isrForCrosstalkSources_1125895_16.stderr parsl.dataflow.dflow INFO: Task 1844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95f68bbff2c5e143e3e4c330ae169748 with result from task 1844 parsl.dataflow.dflow DEBUG: Task 1844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/34e00344-4513-4591-8448-4080723802d5_isrForCrosstalkSources_1125925_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/34e00344-4513-4591-8448-4080723802d5_isrForCrosstalkSources_1125925_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:21 proc_lsst.multi:146 [INFO] found job 16896642 in provider checkpoint proc_lsst.multi INFO: found job 16896642 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0cec5727a658bd9ee2c74b9dd7f6977 with result from task 1869 parsl.dataflow.dflow DEBUG: Task 1869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/2fec0063-3851-4ddb-9be0-ad5a2e7280b4_isrForCrosstalkSources_1125912_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/2fec0063-3851-4ddb-9be0-ad5a2e7280b4_isrForCrosstalkSources_1125912_16.stderr parsl.dataflow.dflow INFO: Task 1633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6da6d05c35ec4c4f9444ddd99df3b4ad with result from task 1633 parsl.dataflow.dflow DEBUG: Task 1633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/9724aa6f-0f4e-4a46-b523-a63d829efb66_isrForCrosstalkSources_1125920_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/9724aa6f-0f4e-4a46-b523-a63d829efb66_isrForCrosstalkSources_1125920_62.stderr parsl.dataflow.dflow INFO: Task 1650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd537fec66c7e7d79c68d9890d505c29 with result from task 1650 parsl.dataflow.dflow DEBUG: Task 1650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/88602e0f-db3a-4ba5-8335-8bfc2d12d0ab_isrForCrosstalkSources_1125932_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/88602e0f-db3a-4ba5-8335-8bfc2d12d0ab_isrForCrosstalkSources_1125932_60.stderr parsl.dataflow.dflow INFO: Task 1830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5b01765e404a440da30a9e526f840cd with result from task 1830 parsl.dataflow.dflow DEBUG: Task 1830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/892b508a-c1e5-4845-a43a-ac2efb488d03_isrForCrosstalkSources_1125936_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/892b508a-c1e5-4845-a43a-ac2efb488d03_isrForCrosstalkSources_1125936_12.stderr parsl.dataflow.dflow INFO: Task 1824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a31768b4dc581ae3efeccc435fd6c6bb with result from task 1824 parsl.dataflow.dflow DEBUG: Task 1824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/dec41bc0-b3df-4122-b0f4-283800d985f7_isrForCrosstalkSources_1125895_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/dec41bc0-b3df-4122-b0f4-283800d985f7_isrForCrosstalkSources_1125895_18.stderr parsl.dataflow.dflow INFO: Task 1792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c71ebcd9b3ae12e06a46aa0dc8d6ba2e with result from task 1792 parsl.dataflow.dflow DEBUG: Task 1792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/0827ffb1-eed8-42f7-ad5e-bd70053d574c_isrForCrosstalkSources_1125924_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/0827ffb1-eed8-42f7-ad5e-bd70053d574c_isrForCrosstalkSources_1125924_36.stderr parsl.dataflow.dflow INFO: Task 1823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6870294a5524fe488ee725ec82aa7f1 with result from task 1823 parsl.dataflow.dflow DEBUG: Task 1823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/e970b819-165b-4d62-8668-771955b4e3f4_isrForCrosstalkSources_1125904_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/e970b819-165b-4d62-8668-771955b4e3f4_isrForCrosstalkSources_1125904_17.stderr parsl.dataflow.dflow INFO: Task 1733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a846f5370b9c15c3ba4a4903be22a5e8 with result from task 1733 parsl.dataflow.dflow DEBUG: Task 1733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/69bcabfa-2d83-4af6-ac4e-ce6de58db6dc_isrForCrosstalkSources_1125926_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/69bcabfa-2d83-4af6-ac4e-ce6de58db6dc_isrForCrosstalkSources_1125926_50.stderr parsl.dataflow.dflow INFO: Task 1831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eef30d20f9bd9cd112857c846e3db8c6 with result from task 1831 parsl.dataflow.dflow DEBUG: Task 1831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/5824174b-7b6c-41ed-9faa-ff50c00b3fd2_isrForCrosstalkSources_1125945_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/5824174b-7b6c-41ed-9faa-ff50c00b3fd2_isrForCrosstalkSources_1125945_5.stderr parsl.dataflow.dflow INFO: Task 1790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbec8d06c5d087239b8806c8ca4c9a62 with result from task 1790 parsl.dataflow.dflow DEBUG: Task 1790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2d7720a8-7059-45ca-a86e-fc95e03ba0ab_isrForCrosstalkSources_1125895_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2d7720a8-7059-45ca-a86e-fc95e03ba0ab_isrForCrosstalkSources_1125895_2.stderr parsl.dataflow.dflow INFO: Task 1676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9abc9a768eb0634ae6bc09858a0df01 with result from task 1676 parsl.dataflow.dflow DEBUG: Task 1676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/dba6f628-5793-4a24-907f-9bd84bc5bcdb_isrForCrosstalkSources_1125919_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/dba6f628-5793-4a24-907f-9bd84bc5bcdb_isrForCrosstalkSources_1125919_1.stderr parsl.dataflow.dflow INFO: Task 1669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eea0b1d465362854a08c28b8e0e3f9e1 with result from task 1669 parsl.dataflow.dflow DEBUG: Task 1669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/2461f979-5745-4024-ab7f-a0708c310cec_isrForCrosstalkSources_1125910_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/2461f979-5745-4024-ab7f-a0708c310cec_isrForCrosstalkSources_1125910_2.stderr parsl.dataflow.dflow INFO: Task 1746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4974ac446c1967707f9dcfd9962735af with result from task 1746 parsl.dataflow.dflow DEBUG: Task 1746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/0575972c-bb80-47b4-8ddd-c96ebcb0511f_isrForCrosstalkSources_1125926_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/0575972c-bb80-47b4-8ddd-c96ebcb0511f_isrForCrosstalkSources_1125926_54.stderr parsl.dataflow.dflow INFO: Task 1822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dd787eb526d97e83645f0367899e26c with result from task 1822 parsl.dataflow.dflow DEBUG: Task 1822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ffef047d-6248-4984-9e75-230298db02dc_isrForCrosstalkSources_1125947_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ffef047d-6248-4984-9e75-230298db02dc_isrForCrosstalkSources_1125947_31.stderr parsl.dataflow.dflow INFO: Task 1865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aadc0173411415d6059d2948a825e1c4 with result from task 1865 parsl.dataflow.dflow DEBUG: Task 1865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/f731082d-e620-4148-841b-ad979df5050e_isrForCrosstalkSources_1125936_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/f731082d-e620-4148-841b-ad979df5050e_isrForCrosstalkSources_1125936_48.stderr parsl.dataflow.dflow INFO: Task 1868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3a69aeecd33574f20b27185ee635c2a with result from task 1868 parsl.dataflow.dflow DEBUG: Task 1868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/40e3cc81-274c-4ed1-b77e-28df8aef78fe_isrForCrosstalkSources_1125927_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/40e3cc81-274c-4ed1-b77e-28df8aef78fe_isrForCrosstalkSources_1125927_29.stderr parsl.dataflow.dflow INFO: Task 1870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87ba3aa769f6c16aacced27abd2c40d2 with result from task 1870 parsl.dataflow.dflow DEBUG: Task 1870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/7108011b-f11c-4e25-9eab-0fe9d00b9e54_isrForCrosstalkSources_1125936_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/7108011b-f11c-4e25-9eab-0fe9d00b9e54_isrForCrosstalkSources_1125936_60.stderr parsl.dataflow.dflow INFO: Task 1675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c3f8415f29a9cc58f092647d9126216 with result from task 1675 parsl.dataflow.dflow DEBUG: Task 1675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/fda437a3-6967-4c87-aa67-9753e6fe89b6_isrForCrosstalkSources_1125900_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/fda437a3-6967-4c87-aa67-9753e6fe89b6_isrForCrosstalkSources_1125900_11.stderr parsl.dataflow.dflow INFO: Task 1716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2eff124752324959b05877ef925faa9 with result from task 1716 parsl.dataflow.dflow DEBUG: Task 1716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/ef49f33f-5f58-4e38-863e-b7d96a44dc90_isrForCrosstalkSources_1125898_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/ef49f33f-5f58-4e38-863e-b7d96a44dc90_isrForCrosstalkSources_1125898_30.stderr parsl.dataflow.dflow INFO: Task 1834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f9f5f8c9f4236b54ad3538f98b9f253 with result from task 1834 parsl.dataflow.dflow DEBUG: Task 1834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/955efb03-8de2-42f9-b4d4-0b96fa087768_isrForCrosstalkSources_1125939_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/955efb03-8de2-42f9-b4d4-0b96fa087768_isrForCrosstalkSources_1125939_42.stderr parsl.dataflow.dflow INFO: Task 1777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbde2ccba516fd4eb62f81217dc639f5 with result from task 1777 parsl.dataflow.dflow DEBUG: Task 1777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/41ce4bee-b835-4171-b2a6-ba68b98c835a_isrForCrosstalkSources_1125897_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/41ce4bee-b835-4171-b2a6-ba68b98c835a_isrForCrosstalkSources_1125897_55.stderr parsl.dataflow.dflow INFO: Task 1678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d24f79527de7bca1ed8d7ddeae836d3 with result from task 1678 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:22 proc_lsst.multi:146 [INFO] found job 16896643 in provider checkpoint proc_lsst.multi INFO: found job 16896643 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/1f5fc431-705c-4724-a7d4-510a4df43510_isrForCrosstalkSources_1125901_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/1f5fc431-705c-4724-a7d4-510a4df43510_isrForCrosstalkSources_1125901_50.stderr parsl.dataflow.dflow INFO: Task 1681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 645a62fe97885ce82641f1281194dc9c with result from task 1681 parsl.dataflow.dflow DEBUG: Task 1681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/7f0929bf-d86a-4d52-a2e4-6a3302aec38e_isrForCrosstalkSources_1125889_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/7f0929bf-d86a-4d52-a2e4-6a3302aec38e_isrForCrosstalkSources_1125889_43.stderr parsl.dataflow.dflow INFO: Task 1747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36b9b950608f1f165788bdadd25da155 with result from task 1747 parsl.dataflow.dflow DEBUG: Task 1747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/587a884d-f757-4662-ab2a-744174c93b9e_isrForCrosstalkSources_1125912_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/587a884d-f757-4662-ab2a-744174c93b9e_isrForCrosstalkSources_1125912_54.stderr parsl.dataflow.dflow INFO: Task 1832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1070a1637a70be24cd84e504e3d72acb with result from task 1832 parsl.dataflow.dflow DEBUG: Task 1832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/ea81b2c9-a834-4a79-a0c6-f9a4346f3b4f_isrForCrosstalkSources_1125927_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/ea81b2c9-a834-4a79-a0c6-f9a4346f3b4f_isrForCrosstalkSources_1125927_49.stderr parsl.dataflow.dflow INFO: Task 1867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08bdea1e6f70238821313168eeea8fb7 with result from task 1867 parsl.dataflow.dflow DEBUG: Task 1867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a87b5452-2885-45c2-a17b-43e620e7e158_isrForCrosstalkSources_1125938_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a87b5452-2885-45c2-a17b-43e620e7e158_isrForCrosstalkSources_1125938_23.stderr parsl.dataflow.dflow INFO: Task 1682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1a9cfeb01e15a3be70647433c52065e with result from task 1682 parsl.dataflow.dflow DEBUG: Task 1682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/336eab5d-4ce1-4c6a-9964-4a29a86baae2_isrForCrosstalkSources_1125921_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/336eab5d-4ce1-4c6a-9964-4a29a86baae2_isrForCrosstalkSources_1125921_42.stderr parsl.dataflow.dflow INFO: Task 1757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 656ec23df3c0fa03dae07783f8b2f2a7 with result from task 1757 parsl.dataflow.dflow DEBUG: Task 1757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/cd5e0332-7212-40a9-94a0-9a96fc95c958_isrForCrosstalkSources_1125923_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/cd5e0332-7212-40a9-94a0-9a96fc95c958_isrForCrosstalkSources_1125923_41.stderr parsl.dataflow.dflow INFO: Task 1845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8e32a3bbf4996c6c24d494544ada135 with result from task 1845 parsl.dataflow.dflow DEBUG: Task 1845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/6cbb2b52-5fc9-4381-9a00-d66716383348_isrForCrosstalkSources_1125908_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/6cbb2b52-5fc9-4381-9a00-d66716383348_isrForCrosstalkSources_1125908_59.stderr parsl.dataflow.dflow INFO: Task 1770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03086eaa0992a822ee33ae9692183139 with result from task 1770 parsl.dataflow.dflow DEBUG: Task 1770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/4131f3ee-5563-4808-92f4-e2483b3e4e9a_isrForCrosstalkSources_1125897_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/4131f3ee-5563-4808-92f4-e2483b3e4e9a_isrForCrosstalkSources_1125897_58.stderr parsl.dataflow.dflow INFO: Task 1784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99088b939bf8462d3a25524c1300cd66 with result from task 1784 parsl.dataflow.dflow DEBUG: Task 1784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/dc5ab3c5-0931-4614-9189-6612e616be24_isrForCrosstalkSources_1125947_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/dc5ab3c5-0931-4614-9189-6612e616be24_isrForCrosstalkSources_1125947_42.stderr parsl.dataflow.dflow INFO: Task 1672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba229739b008cbb7f169a99a386f35a6 with result from task 1672 parsl.dataflow.dflow DEBUG: Task 1672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/ce408f77-801f-42b3-b9eb-1c6b8ad8b852_isrForCrosstalkSources_1125924_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/ce408f77-801f-42b3-b9eb-1c6b8ad8b852_isrForCrosstalkSources_1125924_42.stderr parsl.dataflow.dflow INFO: Task 1753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da8f242a5d12262b3495f40b3028ef50 with result from task 1753 parsl.dataflow.dflow DEBUG: Task 1753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/1c4aa3ed-7800-48da-b2cb-9264048a67f6_isrForCrosstalkSources_1125923_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/1c4aa3ed-7800-48da-b2cb-9264048a67f6_isrForCrosstalkSources_1125923_57.stderr parsl.dataflow.dflow INFO: Task 1762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0119603ed6834503033fa00d12b9012b with result from task 1762 parsl.dataflow.dflow DEBUG: Task 1762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/dcc2b37b-de12-4ee8-9270-99bdaaa69769_isrForCrosstalkSources_1125898_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/dcc2b37b-de12-4ee8-9270-99bdaaa69769_isrForCrosstalkSources_1125898_60.stderr parsl.dataflow.dflow INFO: Task 1670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5a555b83d94de4cf28c30bd0365395c with result from task 1670 parsl.dataflow.dflow DEBUG: Task 1670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/dd4979d2-589e-4fbb-8853-9cd585cd100b_isrForCrosstalkSources_1125941_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/dd4979d2-589e-4fbb-8853-9cd585cd100b_isrForCrosstalkSources_1125941_44.stderr parsl.dataflow.dflow INFO: Task 1773 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06909f1f9c2a7bb3ce4866a2dcf6f3d8 with result from task 1773 parsl.dataflow.dflow DEBUG: Task 1773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/7e5433e0-8aa2-429d-a6a8-1bd0993f39e5_isrForCrosstalkSources_1125896_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/7e5433e0-8aa2-429d-a6a8-1bd0993f39e5_isrForCrosstalkSources_1125896_61.stderr parsl.dataflow.dflow INFO: Task 1657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8d07c755dae525cfa97b931bfd512ce with result from task 1657 parsl.dataflow.dflow DEBUG: Task 1657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a2380059-a33b-423b-8271-35238e54dc02_isrForCrosstalkSources_1125909_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a2380059-a33b-423b-8271-35238e54dc02_isrForCrosstalkSources_1125909_53.stderr parsl.dataflow.dflow INFO: Task 1641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d775df7970853eafe47e9634cf78b1f9 with result from task 1641 parsl.dataflow.dflow DEBUG: Task 1641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/f1ad5966-6720-4a3f-8ee4-ba194ee37bae_isrForCrosstalkSources_1125947_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/f1ad5966-6720-4a3f-8ee4-ba194ee37bae_isrForCrosstalkSources_1125947_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:22 proc_lsst.multi:146 [INFO] found job 16896644 in provider checkpoint proc_lsst.multi INFO: found job 16896644 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2601d7954fb1669587ce8d9599679e29 with result from task 1677 parsl.dataflow.dflow DEBUG: Task 1677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/9a86ce9e-85cd-46d9-8d03-fa834b2df749_isrForCrosstalkSources_1125907_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/9a86ce9e-85cd-46d9-8d03-fa834b2df749_isrForCrosstalkSources_1125907_50.stderr parsl.dataflow.dflow INFO: Task 1739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 708b15dfbef75c2a72c63c46667e20ce with result from task 1739 parsl.dataflow.dflow DEBUG: Task 1739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/b6c681bd-80ab-4867-9f64-0d5496516b4f_isrForCrosstalkSources_1125933_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/b6c681bd-80ab-4867-9f64-0d5496516b4f_isrForCrosstalkSources_1125933_57.stderr parsl.dataflow.dflow INFO: Task 1756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87af097fe4c6d5efb970ac6ab076c421 with result from task 1756 parsl.dataflow.dflow DEBUG: Task 1756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/a0dbb88b-772d-4749-9aaa-29774a7c1461_isrForCrosstalkSources_1125890_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/a0dbb88b-772d-4749-9aaa-29774a7c1461_isrForCrosstalkSources_1125890_60.stderr parsl.dataflow.dflow INFO: Task 1748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57cd7428125de8b99e0aeaa9cb6d8c6e with result from task 1748 parsl.dataflow.dflow DEBUG: Task 1748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/dc3fa744-afcb-48a0-a711-7edf537a098f_isrForCrosstalkSources_1125893_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/dc3fa744-afcb-48a0-a711-7edf537a098f_isrForCrosstalkSources_1125893_44.stderr parsl.dataflow.dflow INFO: Task 1715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7793a05f3cddf96f55cf66e7c387a04 with result from task 1715 parsl.dataflow.dflow DEBUG: Task 1715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/94751682-4b57-4990-a466-30a2385561e1_isrForCrosstalkSources_1125919_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/94751682-4b57-4990-a466-30a2385561e1_isrForCrosstalkSources_1125919_50.stderr parsl.dataflow.dflow INFO: Task 1673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8cb1ce5ff55fabd4314e70cb6cb82f4 with result from task 1673 parsl.dataflow.dflow DEBUG: Task 1673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/8e49d935-6ca0-4c0e-b089-8388841a2d91_isrForCrosstalkSources_1125945_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/8e49d935-6ca0-4c0e-b089-8388841a2d91_isrForCrosstalkSources_1125945_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:23 proc_lsst.multi:146 [INFO] found job 16896645 in provider checkpoint proc_lsst.multi INFO: found job 16896645 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c97eb431be0df7f9e461097dd622b663 with result from task 1714 parsl.dataflow.dflow DEBUG: Task 1714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/3a6082fe-0dff-4be8-a0ee-f532c13f8f37_isrForCrosstalkSources_1125889_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/3a6082fe-0dff-4be8-a0ee-f532c13f8f37_isrForCrosstalkSources_1125889_53.stderr parsl.dataflow.dflow INFO: Task 1835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cddf9811731e2fa3b03cec3e7af6baf with result from task 1835 parsl.dataflow.dflow DEBUG: Task 1835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/badd2abe-0625-4aff-80fb-a0ef72cb9731_isrForCrosstalkSources_1125901_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/badd2abe-0625-4aff-80fb-a0ef72cb9731_isrForCrosstalkSources_1125901_38.stderr parsl.dataflow.dflow INFO: Task 1713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95e0fa4b5b072c027e35e44845e8e355 with result from task 1713 parsl.dataflow.dflow DEBUG: Task 1713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/7b12dde5-7387-4bad-b942-647d64e630f7_isrForCrosstalkSources_1125941_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/7b12dde5-7387-4bad-b942-647d64e630f7_isrForCrosstalkSources_1125941_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:23 proc_lsst.multi:146 [INFO] found job 16896646 in provider checkpoint proc_lsst.multi INFO: found job 16896646 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:23 proc_lsst.multi:146 [INFO] found job 16896647 in provider checkpoint proc_lsst.multi INFO: found job 16896647 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:24 proc_lsst.multi:146 [INFO] found job 16896648 in provider checkpoint proc_lsst.multi INFO: found job 16896648 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:24 proc_lsst.multi:146 [INFO] found job 16896649 in provider checkpoint proc_lsst.multi INFO: found job 16896649 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74e1eabf74312ae98369a99f0ff136fe with result from task 1794 parsl.dataflow.dflow DEBUG: Task 1794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/4f75c51b-7e72-4df2-b996-f99662209f69_isrForCrosstalkSources_1125930_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/4f75c51b-7e72-4df2-b996-f99662209f69_isrForCrosstalkSources_1125930_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:24 proc_lsst.multi:146 [INFO] found job 16896650 in provider checkpoint proc_lsst.multi INFO: found job 16896650 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:24 proc_lsst.multi:146 [INFO] found job 16896651 in provider checkpoint proc_lsst.multi INFO: found job 16896651 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbe46a946bb6d5efc44dcdadd3688d8c with result from task 1891 parsl.dataflow.dflow DEBUG: Task 1891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/b3380d55-31d0-4396-8f8a-3aba126952ec_isrForCrosstalkSources_1125888_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/b3380d55-31d0-4396-8f8a-3aba126952ec_isrForCrosstalkSources_1125888_24.stderr parsl.dataflow.dflow INFO: Task 1902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2de96118f2560f584a5e4c2402d6c92 with result from task 1902 parsl.dataflow.dflow DEBUG: Task 1902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/68c762ca-d5f3-4474-b72e-a9bc0732ac23_isrForCrosstalkSources_1125939_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/68c762ca-d5f3-4474-b72e-a9bc0732ac23_isrForCrosstalkSources_1125939_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:25 proc_lsst.multi:146 [INFO] found job 16896652 in provider checkpoint proc_lsst.multi INFO: found job 16896652 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:25 proc_lsst.multi:146 [INFO] found job 16896653 in provider checkpoint proc_lsst.multi INFO: found job 16896653 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb1f78c445e25f85be199c4ee5acdbfc with result from task 1889 parsl.dataflow.dflow DEBUG: Task 1889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/63b2ff4f-05c9-4356-96f5-7174a4a3f4fe_isrForCrosstalkSources_1125930_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/63b2ff4f-05c9-4356-96f5-7174a4a3f4fe_isrForCrosstalkSources_1125930_41.stderr parsl.dataflow.dflow INFO: Task 1908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f58ae14afd5b58b29acaa91b729b9506 with result from task 1908 parsl.dataflow.dflow DEBUG: Task 1908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/5c65be0f-6dfa-49ad-869c-70206107a27c_isrForCrosstalkSources_1125914_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/5c65be0f-6dfa-49ad-869c-70206107a27c_isrForCrosstalkSources_1125914_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:25 proc_lsst.multi:146 [INFO] found job 16896654 in provider checkpoint proc_lsst.multi INFO: found job 16896654 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d875cef97c4793fac5eba3dd250162dd with result from task 1904 parsl.dataflow.dflow DEBUG: Task 1904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/6abcca8d-c133-44c0-aee2-18b28de189d1_isrForCrosstalkSources_1125893_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/6abcca8d-c133-44c0-aee2-18b28de189d1_isrForCrosstalkSources_1125893_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:25 proc_lsst.multi:146 [INFO] found job 16896655 in provider checkpoint proc_lsst.multi INFO: found job 16896655 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86723f2a60b27a0413a66d2f64a7ebdd with result from task 1906 parsl.dataflow.dflow DEBUG: Task 1906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/332f5514-d9c6-49c1-a027-74a2d867a1e0_isrForCrosstalkSources_1125929_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/332f5514-d9c6-49c1-a027-74a2d867a1e0_isrForCrosstalkSources_1125929_4.stderr parsl.dataflow.dflow INFO: Task 1882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d055dc2f172c227b875790f498d72789 with result from task 1882 parsl.dataflow.dflow DEBUG: Task 1882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/cded9999-9913-4d80-b6b3-7a5932a9c386_isrForCrosstalkSources_1125914_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/cded9999-9913-4d80-b6b3-7a5932a9c386_isrForCrosstalkSources_1125914_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:25 proc_lsst.multi:146 [INFO] found job 16896656 in provider checkpoint proc_lsst.multi INFO: found job 16896656 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcdf9814a9a246e7000817851ab86a8c with result from task 1812 parsl.dataflow.dflow DEBUG: Task 1812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/7443d8bc-64c1-4e67-ad76-fee859f21bcb_isrForCrosstalkSources_1125938_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/7443d8bc-64c1-4e67-ad76-fee859f21bcb_isrForCrosstalkSources_1125938_29.stderr parsl.dataflow.dflow INFO: Task 1732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f5abe512caaee57fae6903c260c1ddf with result from task 1732 parsl.dataflow.dflow DEBUG: Task 1732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/274774da-d7de-48e0-a5eb-595ccbd3079a_isrForCrosstalkSources_1125916_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/274774da-d7de-48e0-a5eb-595ccbd3079a_isrForCrosstalkSources_1125916_23.stderr parsl.dataflow.dflow INFO: Task 1887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 999186acd41a0c09e3ab3328c82e3483 with result from task 1887 parsl.dataflow.dflow DEBUG: Task 1887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/fa547d91-ce4a-4736-b11b-11869878d5d3_isrForCrosstalkSources_1125914_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/fa547d91-ce4a-4736-b11b-11869878d5d3_isrForCrosstalkSources_1125914_58.stderr parsl.dataflow.dflow INFO: Task 1856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f988dfc007dd86bbc5d84d82bf42e613 with result from task 1856 parsl.dataflow.dflow DEBUG: Task 1856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9654a7d4-48ae-49bc-94fc-fee923ef6175_isrForCrosstalkSources_1125921_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9654a7d4-48ae-49bc-94fc-fee923ef6175_isrForCrosstalkSources_1125921_45.stderr parsl.dataflow.dflow INFO: Task 1929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 535c49ef46926abc40615980527a56df with result from task 1929 parsl.dataflow.dflow DEBUG: Task 1929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/2c81bcc6-c907-4b99-a4c3-47ddadfb5360_isrForCrosstalkSources_1125903_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/2c81bcc6-c907-4b99-a4c3-47ddadfb5360_isrForCrosstalkSources_1125903_26.stderr parsl.dataflow.dflow INFO: Task 1966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d94ba56acd562dfa4b13cccd9495d86 with result from task 1966 parsl.dataflow.dflow DEBUG: Task 1966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/f6281314-5726-4d82-9c3d-4c137269a434_isrForCrosstalkSources_1125947_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/f6281314-5726-4d82-9c3d-4c137269a434_isrForCrosstalkSources_1125947_27.stderr parsl.dataflow.dflow INFO: Task 1915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 775e68e2cd94a456a485a282a0308cf9 with result from task 1915 parsl.dataflow.dflow DEBUG: Task 1915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e406a09e-0d7e-464f-8a84-914499a73d55_isrForCrosstalkSources_1125942_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e406a09e-0d7e-464f-8a84-914499a73d55_isrForCrosstalkSources_1125942_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 1933 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 577978f2284501dcea8f4710be3d451a with result from task 1933 parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 1933 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 1933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/5fcabf08-6be7-465c-a3f2-cff56a99bd43_isrForCrosstalkSources_1125901_23.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 1933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/5fcabf08-6be7-465c-a3f2-cff56a99bd43_isrForCrosstalkSources_1125901_23.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:26 proc_lsst.multi:146 [INFO] found job 16896657 in provider checkpoint proc_lsst.multi INFO: found job 16896657 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6aae1ea6fd87f9964ce00f59bc5533ee with result from task 1916 parsl.dataflow.dflow DEBUG: Task 1916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/3496d3c5-9663-4eb5-94c3-f8dc9e19ba53_isrForCrosstalkSources_1125890_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/3496d3c5-9663-4eb5-94c3-f8dc9e19ba53_isrForCrosstalkSources_1125890_40.stderr parsl.dataflow.dflow INFO: Task 1900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5994e4f19b0b7e23f923f6231b0fb0f3 with result from task 1900 parsl.dataflow.dflow DEBUG: Task 1900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e4f92003-31b1-44f8-95d5-228a720327e0_isrForCrosstalkSources_1125901_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e4f92003-31b1-44f8-95d5-228a720327e0_isrForCrosstalkSources_1125901_41.stderr parsl.dataflow.dflow INFO: Task 1852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d0e520cfb16fcc5b7b8e86c885d3305 with result from task 1852 parsl.dataflow.dflow DEBUG: Task 1852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/4bbc8ec1-ad1d-41d5-a8c9-5642c4a140b4_isrForCrosstalkSources_1125923_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/4bbc8ec1-ad1d-41d5-a8c9-5642c4a140b4_isrForCrosstalkSources_1125923_24.stderr parsl.dataflow.dflow INFO: Task 1945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69bad61032e477c42078b3ed2180b959 with result from task 1945 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:26 proc_lsst.multi:146 [INFO] found job 16896658 in provider checkpoint proc_lsst.multi INFO: found job 16896658 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/cd6d3c23-b37c-45f7-9b0e-753938d5006b_isrForCrosstalkSources_1125914_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/cd6d3c23-b37c-45f7-9b0e-753938d5006b_isrForCrosstalkSources_1125914_10.stderr parsl.dataflow.dflow INFO: Task 1846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c013c3f6fad8af934cf79811ab4a7fc with result from task 1846 parsl.dataflow.dflow DEBUG: Task 1846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/af1d8b5d-ca49-4d6c-8c21-28a273bc23e0_isrForCrosstalkSources_1125896_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/af1d8b5d-ca49-4d6c-8c21-28a273bc23e0_isrForCrosstalkSources_1125896_20.stderr parsl.dataflow.dflow INFO: Task 1907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 404240f0149d103f544ecd9cc1a70acf with result from task 1907 parsl.dataflow.dflow DEBUG: Task 1907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/f5946148-c50f-4946-a709-ef5b94390588_isrForCrosstalkSources_1125889_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/f5946148-c50f-4946-a709-ef5b94390588_isrForCrosstalkSources_1125889_36.stderr parsl.dataflow.dflow INFO: Task 1937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47897892e917ef1076e8320e3ce74dfa with result from task 1937 parsl.dataflow.dflow DEBUG: Task 1937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/55924b7a-540a-4aac-b0a2-51317a5d1aca_isrForCrosstalkSources_1125944_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/55924b7a-540a-4aac-b0a2-51317a5d1aca_isrForCrosstalkSources_1125944_40.stderr parsl.dataflow.dflow INFO: Task 1862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2e6bfbaab490e5a2922c3863356de77 with result from task 1862 parsl.dataflow.dflow DEBUG: Task 1862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/f4c6159e-2c84-4e91-a192-9b5e04183ac0_isrForCrosstalkSources_1125925_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/f4c6159e-2c84-4e91-a192-9b5e04183ac0_isrForCrosstalkSources_1125925_56.stderr parsl.dataflow.dflow INFO: Task 1910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3d4ea20a35f53bf7dedb4880a3cb401 with result from task 1910 parsl.dataflow.dflow DEBUG: Task 1910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/cadf1e08-d7d5-4e37-a34d-bf4eed63de15_isrForCrosstalkSources_1125940_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/cadf1e08-d7d5-4e37-a34d-bf4eed63de15_isrForCrosstalkSources_1125940_31.stderr parsl.dataflow.dflow INFO: Task 1783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47abc9729bea2beeb691ffed3bbf9666 with result from task 1783 parsl.dataflow.dflow DEBUG: Task 1783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/27aa430e-4282-49a3-8024-b056b6ae21d6_isrForCrosstalkSources_1125947_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/27aa430e-4282-49a3-8024-b056b6ae21d6_isrForCrosstalkSources_1125947_21.stderr parsl.dataflow.dflow INFO: Task 1936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8296ea10b1870bb227514ad22aa93507 with result from task 1936 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 1936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/b1d39a75-8fd5-4527-a512-222dab1046da_isrForCrosstalkSources_1125915_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/b1d39a75-8fd5-4527-a512-222dab1046da_isrForCrosstalkSources_1125915_49.stderr parsl.dataflow.dflow INFO: Task 1886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cddcd6019b988b3cba72299e8289839 with result from task 1886 parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:26 proc_lsst.multi:146 [INFO] found job 16896659 in provider checkpoint proc_lsst.multi INFO: found job 16896659 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 1886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/d0f85e85-2a0e-4306-b63a-b550e4ff375b_isrForCrosstalkSources_1125921_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/d0f85e85-2a0e-4306-b63a-b550e4ff375b_isrForCrosstalkSources_1125921_55.stderr parsl.dataflow.dflow INFO: Task 1884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6787da549ff5c681ed8fa210a38fe081 with result from task 1884 parsl.dataflow.dflow DEBUG: Task 1884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/92fe8a36-5e86-4164-98c3-34d2e9424478_isrForCrosstalkSources_1125910_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/92fe8a36-5e86-4164-98c3-34d2e9424478_isrForCrosstalkSources_1125910_60.stderr parsl.dataflow.dflow INFO: Task 1809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34ad81e4cbc0ff0492a7d528d05949f4 with result from task 1809 parsl.dataflow.dflow DEBUG: Task 1809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/57e3b4b8-4390-4d5e-9265-ac720bdc2419_isrForCrosstalkSources_1125902_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/57e3b4b8-4390-4d5e-9265-ac720bdc2419_isrForCrosstalkSources_1125902_60.stderr parsl.dataflow.dflow INFO: Task 1858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97d459189abc0b8e91361fe679d9c582 with result from task 1858 parsl.dataflow.dflow DEBUG: Task 1858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/c36309e8-d5c0-4428-9201-7f62d6914e0c_isrForCrosstalkSources_1125908_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/c36309e8-d5c0-4428-9201-7f62d6914e0c_isrForCrosstalkSources_1125908_15.stderr parsl.dataflow.dflow INFO: Task 1947 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ede1170f34561d18a496936bf43d746 with result from task 1947 parsl.dataflow.dflow DEBUG: Task 1947 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/b02eeb17-4423-49a2-b07a-39fd62b3a31c_isrForCrosstalkSources_1125914_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/b02eeb17-4423-49a2-b07a-39fd62b3a31c_isrForCrosstalkSources_1125914_23.stderr parsl.dataflow.dflow INFO: Task 1954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26c719b708415a03ae29fe02fbd163b4 with result from task 1954 parsl.dataflow.dflow DEBUG: Task 1954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/297dbfd2-fdf4-4ae2-854d-8645cd02594c_isrForCrosstalkSources_1125947_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/297dbfd2-fdf4-4ae2-854d-8645cd02594c_isrForCrosstalkSources_1125947_12.stderr parsl.dataflow.dflow INFO: Task 1955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd4ef0c7a265373b083e944ac6c3a9ad with result from task 1955 parsl.dataflow.dflow DEBUG: Task 1955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/a22a64f1-868e-45f0-90e2-2dce667ca0bd_isrForCrosstalkSources_1125894_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/a22a64f1-868e-45f0-90e2-2dce667ca0bd_isrForCrosstalkSources_1125894_3.stderr parsl.dataflow.dflow INFO: Task 1877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fdf4f73255e6dcbcc4c0fcb9c4094dc with result from task 1877 parsl.dataflow.dflow DEBUG: Task 1877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/f31c1848-dfbb-4d42-a786-55a6eda7855e_isrForCrosstalkSources_1125920_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/f31c1848-dfbb-4d42-a786-55a6eda7855e_isrForCrosstalkSources_1125920_17.stderr parsl.dataflow.dflow INFO: Task 1895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 467f5c2d2b1495407ab6b2217b508bd8 with result from task 1895 parsl.dataflow.dflow DEBUG: Task 1895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/b7c32181-713a-45fc-b30b-d1adcd507fc4_isrForCrosstalkSources_1125946_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/b7c32181-713a-45fc-b30b-d1adcd507fc4_isrForCrosstalkSources_1125946_9.stderr parsl.dataflow.dflow INFO: Task 1829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ba0e942f24172c5fd2cca7a9505dca3 with result from task 1829 parsl.dataflow.dflow DEBUG: Task 1829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/2f37f18c-ef40-44e7-8c5c-33499aa4a80a_isrForCrosstalkSources_1125915_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/2f37f18c-ef40-44e7-8c5c-33499aa4a80a_isrForCrosstalkSources_1125915_62.stderr parsl.dataflow.dflow INFO: Task 1911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e248d19e97879a80052f1b633525ff9d with result from task 1911 parsl.dataflow.dflow DEBUG: Task 1911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/0abaecf6-4180-4c00-89e4-c878bd385000_isrForCrosstalkSources_1125912_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/0abaecf6-4180-4c00-89e4-c878bd385000_isrForCrosstalkSources_1125912_62.stderr parsl.dataflow.dflow INFO: Task 1943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd804f04252b8315cbbaad123074cfdf with result from task 1943 parsl.dataflow.dflow DEBUG: Task 1943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/942680e4-ae2d-448c-9f3b-b0315b17ebcb_isrForCrosstalkSources_1125889_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/942680e4-ae2d-448c-9f3b-b0315b17ebcb_isrForCrosstalkSources_1125889_11.stderr parsl.dataflow.dflow INFO: Task 1914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8855337801fea3e64f8dcce79ab7854e with result from task 1914 parsl.dataflow.dflow DEBUG: Task 1914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/182e3257-a94c-47d6-bd35-5bf1e048314f_isrForCrosstalkSources_1125937_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/182e3257-a94c-47d6-bd35-5bf1e048314f_isrForCrosstalkSources_1125937_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:27 proc_lsst.multi:146 [INFO] found job 16896660 in provider checkpoint proc_lsst.multi INFO: found job 16896660 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d7fe03d390c1ae15b0ffbb10975f724 with result from task 1984 parsl.dataflow.dflow DEBUG: Task 1984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/31bceb9d-0ed1-476d-8d9f-a2d342f6d533_isrForCrosstalkSources_1125900_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/31bceb9d-0ed1-476d-8d9f-a2d342f6d533_isrForCrosstalkSources_1125900_13.stderr parsl.dataflow.dflow INFO: Task 1982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 052e010d428b7165dc48bd89b0945273 with result from task 1982 parsl.dataflow.dflow DEBUG: Task 1982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/dc1290d0-39d7-4bf1-97fd-3f5de61930de_isrForCrosstalkSources_1125932_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/dc1290d0-39d7-4bf1-97fd-3f5de61930de_isrForCrosstalkSources_1125932_29.stderr parsl.dataflow.dflow INFO: Task 1967 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5df2215cc2b0c1c39a1c13ae0467b687 with result from task 1967 parsl.dataflow.dflow DEBUG: Task 1967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/2bb270cb-d78d-4e44-94b5-53126a3cfe58_isrForCrosstalkSources_1125946_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/2bb270cb-d78d-4e44-94b5-53126a3cfe58_isrForCrosstalkSources_1125946_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:27 proc_lsst.multi:146 [INFO] found job 16896661 in provider checkpoint proc_lsst.multi INFO: found job 16896661 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1e4192b149f5f7d3379245df9115bc5 with result from task 1874 parsl.dataflow.dflow DEBUG: Task 1874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/49a66dd1-95f1-4398-9f88-93bcf3dd72d0_isrForCrosstalkSources_1125942_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/49a66dd1-95f1-4398-9f88-93bcf3dd72d0_isrForCrosstalkSources_1125942_10.stderr parsl.dataflow.dflow INFO: Task 1951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1a37add4119624aec5970a838a152bd with result from task 1951 parsl.dataflow.dflow DEBUG: Task 1951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/9f93ffb0-bfdd-48b4-8748-77d91b7e8a94_isrForCrosstalkSources_1125932_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/9f93ffb0-bfdd-48b4-8748-77d91b7e8a94_isrForCrosstalkSources_1125932_57.stderr parsl.dataflow.dflow INFO: Task 1983 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acc5b79f70393cef5a0dfdd19899b18d with result from task 1983 parsl.dataflow.dflow DEBUG: Task 1983 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/0ae3014f-555a-43a5-8b48-b2b48b8b1376_isrForCrosstalkSources_1125919_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/0ae3014f-555a-43a5-8b48-b2b48b8b1376_isrForCrosstalkSources_1125919_4.stderr parsl.dataflow.dflow INFO: Task 1961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b64a26ae2f85efaebb708fb4cdb3104 with result from task 1961 parsl.dataflow.dflow DEBUG: Task 1961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/812ba751-cf53-4de4-9877-89f73b33612c_isrForCrosstalkSources_1125930_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/812ba751-cf53-4de4-9877-89f73b33612c_isrForCrosstalkSources_1125930_43.stderr parsl.dataflow.dflow INFO: Task 1775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d25c507e0c72a011caf7a9a88eda060 with result from task 1775 parsl.dataflow.dflow DEBUG: Task 1775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/ecc09370-a413-439e-9355-3a96d884b1c5_isrForCrosstalkSources_1125903_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/ecc09370-a413-439e-9355-3a96d884b1c5_isrForCrosstalkSources_1125903_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:27 proc_lsst.multi:146 [INFO] found job 16896662 in provider checkpoint proc_lsst.multi INFO: found job 16896662 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2046 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2567de99283ecf1b05880353debdf08 with result from task 2046 parsl.dataflow.dflow DEBUG: Task 2046 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/b3f5d013-79ca-4d24-9549-92f51e5c47c9_isrForCrosstalkSources_1125909_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/b3f5d013-79ca-4d24-9549-92f51e5c47c9_isrForCrosstalkSources_1125909_11.stderr parsl.dataflow.dflow INFO: Task 1859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99d795608abfd5978fa61d3b50ee55f7 with result from task 1859 parsl.dataflow.dflow DEBUG: Task 1859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/1210ab85-128e-478b-a120-5d5074fbb322_isrForCrosstalkSources_1125888_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/1210ab85-128e-478b-a120-5d5074fbb322_isrForCrosstalkSources_1125888_9.stderr parsl.dataflow.dflow INFO: Task 1962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 783f38efee29c71fb167be4a96fd731f with result from task 1962 parsl.dataflow.dflow DEBUG: Task 1962 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/3ad071c6-4b2e-4ba7-ad80-cc8249457326_isrForCrosstalkSources_1125903_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/3ad071c6-4b2e-4ba7-ad80-cc8249457326_isrForCrosstalkSources_1125903_16.stderr parsl.dataflow.dflow INFO: Task 1934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ff350cb67b9328706d6ecc5d845df2b with result from task 1934 parsl.dataflow.dflow DEBUG: Task 1934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/f27dd882-c714-42f9-90e7-b0dd8f243834_isrForCrosstalkSources_1125902_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/f27dd882-c714-42f9-90e7-b0dd8f243834_isrForCrosstalkSources_1125902_15.stderr parsl.dataflow.dflow INFO: Task 1942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1768b156cee3afe59e2d1612dde9502 with result from task 1942 parsl.dataflow.dflow DEBUG: Task 1942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/0f0d6bae-6fc5-459a-809d-3bc6e465e37e_isrForCrosstalkSources_1125888_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/0f0d6bae-6fc5-459a-809d-3bc6e465e37e_isrForCrosstalkSources_1125888_39.stderr parsl.dataflow.dflow INFO: Task 1997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a2aeaa4af1d30824669c64517a80c33 with result from task 1997 parsl.dataflow.dflow DEBUG: Task 1997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/20b64011-a5a6-46af-8acf-317d17f45b34_isrForCrosstalkSources_1125906_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/20b64011-a5a6-46af-8acf-317d17f45b34_isrForCrosstalkSources_1125906_25.stderr parsl.dataflow.dflow INFO: Task 1941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c96475dcb22255dcff97091353ba6269 with result from task 1941 parsl.dataflow.dflow DEBUG: Task 1941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/61b5e19c-79a1-44b5-8740-67db88170a8f_isrForCrosstalkSources_1125926_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/61b5e19c-79a1-44b5-8740-67db88170a8f_isrForCrosstalkSources_1125926_60.stderr parsl.dataflow.dflow INFO: Task 1905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1d311dbe4c374ae35db8653decafd02 with result from task 1905 parsl.dataflow.dflow DEBUG: Task 1905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/e7e2f45c-bab8-49ae-9f30-4f0ed3b34248_isrForCrosstalkSources_1125906_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/e7e2f45c-bab8-49ae-9f30-4f0ed3b34248_isrForCrosstalkSources_1125906_40.stderr parsl.dataflow.dflow INFO: Task 1981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad3ac02c4efed6b38322061025e706cf with result from task 1981 parsl.dataflow.dflow DEBUG: Task 1981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/adc1474f-55e2-4a65-ab5d-02cecf253d47_isrForCrosstalkSources_1125936_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/adc1474f-55e2-4a65-ab5d-02cecf253d47_isrForCrosstalkSources_1125936_34.stderr parsl.dataflow.dflow INFO: Task 1980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11f18a32f501c3579ef126d6135f011f with result from task 1980 parsl.dataflow.dflow DEBUG: Task 1980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/eeb0f489-a34c-483f-9bcf-696e4f7e6e66_isrForCrosstalkSources_1125890_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/eeb0f489-a34c-483f-9bcf-696e4f7e6e66_isrForCrosstalkSources_1125890_36.stderr parsl.dataflow.dflow INFO: Task 1857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ced38e9c79c8d9c26ea457272a306c9 with result from task 1857 parsl.dataflow.dflow DEBUG: Task 1857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/621aa51e-5dda-46bb-bb6f-48a59bafd98b_isrForCrosstalkSources_1125944_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/621aa51e-5dda-46bb-bb6f-48a59bafd98b_isrForCrosstalkSources_1125944_31.stderr parsl.dataflow.dflow INFO: Task 1864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1088521c8cde1e347b0eadb97fc935d3 with result from task 1864 parsl.dataflow.dflow DEBUG: Task 1864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/126751ff-2bcb-449e-9e45-60fa5abf0a96_isrForCrosstalkSources_1125911_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/126751ff-2bcb-449e-9e45-60fa5abf0a96_isrForCrosstalkSources_1125911_28.stderr parsl.dataflow.dflow INFO: Task 1995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c393279aed869c4ca42a20bd3e8642f with result from task 1995 parsl.dataflow.dflow DEBUG: Task 1995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/0e1f3798-9291-4504-beec-cafced98e88f_isrForCrosstalkSources_1125935_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/0e1f3798-9291-4504-beec-cafced98e88f_isrForCrosstalkSources_1125935_8.stderr parsl.dataflow.dflow INFO: Task 1901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb71ea3c0f63b739c8b248539b6d8bd0 with result from task 1901 parsl.dataflow.dflow DEBUG: Task 1901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/5caa3c1d-9f92-4175-ab09-25182e5fc41b_isrForCrosstalkSources_1125897_34.stdout parsl.dataflow.dflow INFO: Standard error for task 1901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/5caa3c1d-9f92-4175-ab09-25182e5fc41b_isrForCrosstalkSources_1125897_34.stderr parsl.dataflow.dflow INFO: Task 2017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26b083238b356db69ab36646479843b3 with result from task 2017 parsl.dataflow.dflow DEBUG: Task 2017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/3f36ffb5-14e0-42bd-af18-f5c73c1b1ade_isrForCrosstalkSources_1125921_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/3f36ffb5-14e0-42bd-af18-f5c73c1b1ade_isrForCrosstalkSources_1125921_35.stderr parsl.dataflow.dflow INFO: Task 1896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc935401b176a128ebba32a4c488cf75 with result from task 1896 parsl.dataflow.dflow DEBUG: Task 1896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/3123e2cb-0017-4e9e-8b72-028acdea9ad8_isrForCrosstalkSources_1125916_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/3123e2cb-0017-4e9e-8b72-028acdea9ad8_isrForCrosstalkSources_1125916_4.stderr parsl.dataflow.dflow INFO: Task 1993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68bc0210759e0aefcb9e4a5ad1eee94f with result from task 1993 parsl.dataflow.dflow DEBUG: Task 1993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/4d69a4ed-a53e-4668-b311-2c26671e5d5f_isrForCrosstalkSources_1125921_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/4d69a4ed-a53e-4668-b311-2c26671e5d5f_isrForCrosstalkSources_1125921_13.stderr parsl.dataflow.dflow INFO: Task 2029 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b38b36d150e386c70f4659818aab8449 with result from task 2029 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 2029 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/d5ad1027-3646-4e33-be2a-ddb3960c3f22_isrForCrosstalkSources_1125910_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/d5ad1027-3646-4e33-be2a-ddb3960c3f22_isrForCrosstalkSources_1125910_33.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:28 proc_lsst.multi:146 [INFO] found job 16896663 in provider checkpoint proc_lsst.multi INFO: found job 16896663 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46c396dc5665d82321c780103d8a9b50 with result from task 2021 parsl.dataflow.dflow DEBUG: Task 2021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8b742e02-811f-4d7a-aa9d-83a6fb37a4f1_isrForCrosstalkSources_1125891_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8b742e02-811f-4d7a-aa9d-83a6fb37a4f1_isrForCrosstalkSources_1125891_28.stderr parsl.dataflow.dflow INFO: Task 1686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a027e92a99ab01524951b7b72e885e09 with result from task 1686 parsl.dataflow.dflow DEBUG: Task 1686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/59ccff31-3df7-45fe-93d6-df98c3bede3f_isrForCrosstalkSources_1125909_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/59ccff31-3df7-45fe-93d6-df98c3bede3f_isrForCrosstalkSources_1125909_4.stderr parsl.dataflow.dflow INFO: Task 1926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cff172596c595998e747b45b7315c80e with result from task 1926 parsl.dataflow.dflow DEBUG: Task 1926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/5a3b338f-20fc-4393-82fc-6acb18d1f37e_isrForCrosstalkSources_1125939_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/5a3b338f-20fc-4393-82fc-6acb18d1f37e_isrForCrosstalkSources_1125939_47.stderr parsl.dataflow.dflow INFO: Task 1991 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1af090844672e3d9a8e8031213ce5ea5 with result from task 1991 parsl.dataflow.dflow DEBUG: Task 1991 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/e7ec964c-a391-4137-b1a9-e6f20b1c86be_isrForCrosstalkSources_1125905_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/e7ec964c-a391-4137-b1a9-e6f20b1c86be_isrForCrosstalkSources_1125905_24.stderr parsl.dataflow.dflow INFO: Task 1924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d00d4f129c9a007d2069cf39eca84cd8 with result from task 1924 parsl.dataflow.dflow DEBUG: Task 1924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/08b2fa34-409a-4083-8bba-6ce6a8a14c0c_isrForCrosstalkSources_1125894_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/08b2fa34-409a-4083-8bba-6ce6a8a14c0c_isrForCrosstalkSources_1125894_22.stderr parsl.dataflow.dflow INFO: Task 1932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3a82a2cd51ab8ad98563ae94e2fd841 with result from task 1932 parsl.dataflow.dflow DEBUG: Task 1932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/70c9658e-c61c-4f91-b5a7-972290625396_isrForCrosstalkSources_1125938_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/70c9658e-c61c-4f91-b5a7-972290625396_isrForCrosstalkSources_1125938_26.stderr parsl.dataflow.dflow INFO: Task 2024 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d0a4eadadaca20b542348abbd279b08 with result from task 2024 parsl.dataflow.dflow DEBUG: Task 2024 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/52e091b1-2c74-4877-9910-d3c07c846d1d_isrForCrosstalkSources_1125894_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/52e091b1-2c74-4877-9910-d3c07c846d1d_isrForCrosstalkSources_1125894_31.stderr parsl.dataflow.dflow INFO: Task 1863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 881ba9579d94d3f6e036b9f85cb00208 with result from task 1863 parsl.dataflow.dflow DEBUG: Task 1863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/b080f8c3-b4ee-4809-a15a-be48db02ef11_isrForCrosstalkSources_1125930_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/b080f8c3-b4ee-4809-a15a-be48db02ef11_isrForCrosstalkSources_1125930_58.stderr parsl.dataflow.dflow INFO: Task 1774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48ddca6fa233828b0d187a17c59e9f14 with result from task 1774 parsl.dataflow.dflow DEBUG: Task 1774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f518115f-87c9-4bbf-a6c7-e42c309490b9_isrForCrosstalkSources_1125930_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f518115f-87c9-4bbf-a6c7-e42c309490b9_isrForCrosstalkSources_1125930_59.stderr parsl.dataflow.dflow INFO: Task 1776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0956f8b33e16d9c205ad161e8e3c2b9 with result from task 1776 parsl.dataflow.dflow DEBUG: Task 1776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/bc3a1b35-93dc-413f-9004-f8874b61e4c8_isrForCrosstalkSources_1125943_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/bc3a1b35-93dc-413f-9004-f8874b61e4c8_isrForCrosstalkSources_1125943_49.stderr parsl.dataflow.dflow INFO: Task 1972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6095c96cf3a6a99aaf68eb2b8737c4f with result from task 1972 parsl.dataflow.dflow DEBUG: Task 1972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/543498de-133f-4395-b138-a8f1f606b3ca_isrForCrosstalkSources_1125909_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/543498de-133f-4395-b138-a8f1f606b3ca_isrForCrosstalkSources_1125909_52.stderr parsl.dataflow.dflow INFO: Task 1898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbbb3b5808cd443f720e2de8462e4375 with result from task 1898 parsl.dataflow.dflow DEBUG: Task 1898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0f1ba34b-de3e-4326-889c-f32f6eb276a3_isrForCrosstalkSources_1125904_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0f1ba34b-de3e-4326-889c-f32f6eb276a3_isrForCrosstalkSources_1125904_16.stderr parsl.dataflow.dflow INFO: Task 1990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe8effbdbb09dff13df9f78a673d5a2f with result from task 1990 parsl.dataflow.dflow DEBUG: Task 1990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/9c4f426d-2e66-49d3-8475-df1ccb25e1f1_isrForCrosstalkSources_1125942_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/9c4f426d-2e66-49d3-8475-df1ccb25e1f1_isrForCrosstalkSources_1125942_44.stderr parsl.dataflow.dflow INFO: Task 1998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c98f408210545acaa4cf05e3ab5323a with result from task 1998 parsl.dataflow.dflow DEBUG: Task 1998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/bc4c2c09-bfac-4aff-b11e-5450f2183889_isrForCrosstalkSources_1125898_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/bc4c2c09-bfac-4aff-b11e-5450f2183889_isrForCrosstalkSources_1125898_20.stderr parsl.dataflow.dflow INFO: Task 1893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa4ed3873adcb0f51e8b008920f706e5 with result from task 1893 parsl.dataflow.dflow DEBUG: Task 1893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/bb032897-2bcb-4957-9f83-feca7ec0e4e4_isrForCrosstalkSources_1125905_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/bb032897-2bcb-4957-9f83-feca7ec0e4e4_isrForCrosstalkSources_1125905_5.stderr parsl.dataflow.dflow INFO: Task 1878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8001abcc83cb1a251eda50469ff2a24 with result from task 1878 parsl.dataflow.dflow DEBUG: Task 1878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/b2c27efc-68df-4575-9239-70b9e6e983d8_isrForCrosstalkSources_1125929_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/b2c27efc-68df-4575-9239-70b9e6e983d8_isrForCrosstalkSources_1125929_52.stderr parsl.dataflow.dflow INFO: Task 1992 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16704eba95355f46547f923edb3be3dd with result from task 1992 parsl.dataflow.dflow DEBUG: Task 1992 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/4c668d50-56f8-4f77-af65-51531032d4e6_isrForCrosstalkSources_1125890_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/4c668d50-56f8-4f77-af65-51531032d4e6_isrForCrosstalkSources_1125890_30.stderr parsl.dataflow.dflow INFO: Task 1848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65dfa2c9c16f4f51f5d04cedf5410968 with result from task 1848 parsl.dataflow.dflow DEBUG: Task 1848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/dbadbf9c-2d18-4b64-8839-91629bd4b417_isrForCrosstalkSources_1125932_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/dbadbf9c-2d18-4b64-8839-91629bd4b417_isrForCrosstalkSources_1125932_19.stderr parsl.dataflow.dflow INFO: Task 1903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50cb919832dcd8a9f95ee851ecc1ba9c with result from task 1903 parsl.dataflow.dflow DEBUG: Task 1903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/d50d7ff5-b57f-4c51-92b2-46ff5cd74d26_isrForCrosstalkSources_1125888_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/d50d7ff5-b57f-4c51-92b2-46ff5cd74d26_isrForCrosstalkSources_1125888_33.stderr parsl.dataflow.dflow INFO: Task 1956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a196e9accb3a16a0216bf1d849be98e9 with result from task 1956 parsl.dataflow.dflow DEBUG: Task 1956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/034cb056-c9ef-480a-a877-7910e7e69016_isrForCrosstalkSources_1125934_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/034cb056-c9ef-480a-a877-7910e7e69016_isrForCrosstalkSources_1125934_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:28 proc_lsst.multi:146 [INFO] found job 16896664 in provider checkpoint proc_lsst.multi INFO: found job 16896664 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f53b9638dd9d42b6af4e265a8fc8432f with result from task 1986 parsl.dataflow.dflow DEBUG: Task 1986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/a9317c72-079a-40f9-9bb1-65ef93e448b9_isrForCrosstalkSources_1125897_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/a9317c72-079a-40f9-9bb1-65ef93e448b9_isrForCrosstalkSources_1125897_18.stderr parsl.dataflow.dflow INFO: Task 1988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c90338b43c60e1e0dddf5548b58afe5 with result from task 1988 parsl.dataflow.dflow DEBUG: Task 1988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/6790843a-8a5d-4447-9735-38c7c36e2e5c_isrForCrosstalkSources_1125941_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/6790843a-8a5d-4447-9735-38c7c36e2e5c_isrForCrosstalkSources_1125941_43.stderr parsl.dataflow.dflow INFO: Task 1899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f860626ad740df175e7cda9b9a250c8a with result from task 1899 parsl.dataflow.dflow DEBUG: Task 1899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/7fb90f8c-9fc5-49a4-abc2-27576b2ba298_isrForCrosstalkSources_1125901_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/7fb90f8c-9fc5-49a4-abc2-27576b2ba298_isrForCrosstalkSources_1125901_40.stderr parsl.dataflow.dflow INFO: Task 1973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4869cddae430017367ee5708c3da295b with result from task 1973 parsl.dataflow.dflow DEBUG: Task 1973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/10c26f11-161f-4a3e-a013-538b8bb7de93_isrForCrosstalkSources_1125940_6.stdout parsl.dataflow.dflow INFO: Standard error for task 1973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/10c26f11-161f-4a3e-a013-538b8bb7de93_isrForCrosstalkSources_1125940_6.stderr parsl.dataflow.dflow INFO: Task 1964 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53606fca6c5c78a1e33d3e408066a354 with result from task 1964 parsl.dataflow.dflow DEBUG: Task 1964 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/7334268e-baae-439d-aa22-148494159373_isrForCrosstalkSources_1125930_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/7334268e-baae-439d-aa22-148494159373_isrForCrosstalkSources_1125930_56.stderr parsl.dataflow.dflow INFO: Task 2043 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e9f90d5b68feb5861c0293f210266d5 with result from task 2043 parsl.dataflow.dflow DEBUG: Task 2043 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/f655f3c2-9b26-4fee-b7a7-e40f245fb163_isrForCrosstalkSources_1125901_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/f655f3c2-9b26-4fee-b7a7-e40f245fb163_isrForCrosstalkSources_1125901_57.stderr parsl.dataflow.dflow INFO: Task 2036 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb37333f676141ea7f9743d1ae8c93c2 with result from task 2036 parsl.dataflow.dflow DEBUG: Task 2036 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/b7d34be0-9951-4195-bc47-83fd660d6422_isrForCrosstalkSources_1125915_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/b7d34be0-9951-4195-bc47-83fd660d6422_isrForCrosstalkSources_1125915_8.stderr parsl.dataflow.dflow INFO: Task 2016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2044fbfbe3848a1a6c52ce695ef60ad7 with result from task 2016 parsl.dataflow.dflow DEBUG: Task 2016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/7b0c44ae-9635-4d20-ac02-e4d7c824e7b7_isrForCrosstalkSources_1125896_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/7b0c44ae-9635-4d20-ac02-e4d7c824e7b7_isrForCrosstalkSources_1125896_59.stderr parsl.dataflow.dflow INFO: Task 1963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82d0762b97116f309de02e3abbfc5509 with result from task 1963 parsl.dataflow.dflow DEBUG: Task 1963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/4505a721-b8b2-471d-a4e1-aea3017f5773_isrForCrosstalkSources_1125915_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/4505a721-b8b2-471d-a4e1-aea3017f5773_isrForCrosstalkSources_1125915_29.stderr parsl.dataflow.dflow INFO: Task 2012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1830479461c485c0ed7aeb98ed0c269c with result from task 2012 parsl.dataflow.dflow DEBUG: Task 2012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/f56840f4-feb0-44c9-9fcc-6c21cca82a4b_isrForCrosstalkSources_1125892_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/f56840f4-feb0-44c9-9fcc-6c21cca82a4b_isrForCrosstalkSources_1125892_32.stderr parsl.dataflow.dflow INFO: Task 2060 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbe4ee024803f3ef0ce62a8297def6d3 with result from task 2060 parsl.dataflow.dflow DEBUG: Task 2060 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/3dbdcb2a-3e40-4566-b9b7-08775f4385ea_isrForCrosstalkSources_1125903_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/3dbdcb2a-3e40-4566-b9b7-08775f4385ea_isrForCrosstalkSources_1125903_59.stderr parsl.dataflow.dflow INFO: Task 2018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e769380b4d188d255652a821a3a650d1 with result from task 2018 parsl.dataflow.dflow DEBUG: Task 2018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/db88680f-5e1c-403a-9ae6-58af76ee0ed8_isrForCrosstalkSources_1125899_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/db88680f-5e1c-403a-9ae6-58af76ee0ed8_isrForCrosstalkSources_1125899_21.stderr parsl.dataflow.dflow INFO: Task 1927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a51c052e624406e4748591fc6a9d9bfe with result from task 1927 parsl.dataflow.dflow DEBUG: Task 1927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/f337b54d-a436-4d1c-9a51-a0e529580713_isrForCrosstalkSources_1125932_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/f337b54d-a436-4d1c-9a51-a0e529580713_isrForCrosstalkSources_1125932_9.stderr parsl.dataflow.dflow INFO: Task 2009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d21b055137aa36c670642c763413fc6 with result from task 2009 parsl.dataflow.dflow DEBUG: Task 2009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ef782dee-1132-4bf0-bd82-8fe2b765b111_isrForCrosstalkSources_1125918_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ef782dee-1132-4bf0-bd82-8fe2b765b111_isrForCrosstalkSources_1125918_31.stderr parsl.dataflow.dflow INFO: Task 2026 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25ed1468cabbba48ad00a8d88acba50e with result from task 2026 parsl.dataflow.dflow DEBUG: Task 2026 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/0c9d69ba-de35-4765-b9a9-50263915b5c1_isrForCrosstalkSources_1125903_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/0c9d69ba-de35-4765-b9a9-50263915b5c1_isrForCrosstalkSources_1125903_46.stderr parsl.dataflow.dflow INFO: Task 2074 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd393fe4b986a0220f505170040df8ec with result from task 2074 parsl.dataflow.dflow DEBUG: Task 2074 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/223f18db-0f63-4e9d-97b9-503994fc3c55_isrForCrosstalkSources_1125918_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/223f18db-0f63-4e9d-97b9-503994fc3c55_isrForCrosstalkSources_1125918_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:29 proc_lsst.multi:146 [INFO] found job 16896665 in provider checkpoint proc_lsst.multi INFO: found job 16896665 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1950 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 534d41fe24a03cecd865390ad1ee9a28 with result from task 1950 parsl.dataflow.dflow DEBUG: Task 1950 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/ca386bd1-12f8-4f5a-865c-f7a51318f55b_isrForCrosstalkSources_1125925_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/ca386bd1-12f8-4f5a-865c-f7a51318f55b_isrForCrosstalkSources_1125925_14.stderr parsl.dataflow.dflow INFO: Task 2030 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9aac62686e9e3acbb6ed4d3ba7063399 with result from task 2030 parsl.dataflow.dflow DEBUG: Task 2030 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/b0238dea-5353-42bb-b798-64e98757daec_isrForCrosstalkSources_1125941_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/b0238dea-5353-42bb-b798-64e98757daec_isrForCrosstalkSources_1125941_20.stderr parsl.dataflow.dflow INFO: Task 1917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f95140a27526d4c1f2967acbc008f612 with result from task 1917 parsl.dataflow.dflow DEBUG: Task 1917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/eb0d66b7-6b5f-444e-9754-0f03941b2d63_isrForCrosstalkSources_1125943_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/eb0d66b7-6b5f-444e-9754-0f03941b2d63_isrForCrosstalkSources_1125943_23.stderr parsl.dataflow.dflow INFO: Task 2007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b72abacdbd5e86e9e2c4e5b83d928cf7 with result from task 2007 parsl.dataflow.dflow DEBUG: Task 2007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/d3a9b58e-562d-4dcd-a531-b4b3fb370227_isrForCrosstalkSources_1125947_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/d3a9b58e-562d-4dcd-a531-b4b3fb370227_isrForCrosstalkSources_1125947_8.stderr parsl.dataflow.dflow INFO: Task 2048 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ef560034ded3bc53cdc5fe696b37679 with result from task 2048 parsl.dataflow.dflow DEBUG: Task 2048 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/7f871063-4d26-4359-bfeb-3b609ce53633_isrForCrosstalkSources_1125931_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/7f871063-4d26-4359-bfeb-3b609ce53633_isrForCrosstalkSources_1125931_17.stderr parsl.dataflow.dflow INFO: Task 2015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 311dc7c8e406865eb910a7cbd8df8c03 with result from task 2015 parsl.dataflow.dflow DEBUG: Task 2015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/72402885-ef99-45d5-b313-741c0d85a210_isrForCrosstalkSources_1125940_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/72402885-ef99-45d5-b313-741c0d85a210_isrForCrosstalkSources_1125940_30.stderr parsl.dataflow.dflow INFO: Task 2023 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e228ef62583b69ff1871bb78307e9541 with result from task 2023 parsl.dataflow.dflow DEBUG: Task 2023 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/9579481b-836b-4233-b672-f489f8f53f2b_isrForCrosstalkSources_1125904_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/9579481b-836b-4233-b672-f489f8f53f2b_isrForCrosstalkSources_1125904_34.stderr parsl.dataflow.dflow INFO: Task 2014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb83da373b5e01f4797332ac0ddc379a with result from task 2014 parsl.dataflow.dflow DEBUG: Task 2014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/0625331c-6e19-4ecd-964e-9348bdb73a6a_isrForCrosstalkSources_1125932_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/0625331c-6e19-4ecd-964e-9348bdb73a6a_isrForCrosstalkSources_1125932_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:29 proc_lsst.multi:146 [INFO] found job 16896666 in provider checkpoint proc_lsst.multi INFO: found job 16896666 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab07a2efdfea8af469f32636bf0d72de with result from task 2005 parsl.dataflow.dflow DEBUG: Task 2005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d33894fc-dca6-4d19-8307-afc8c1e287c5_isrForCrosstalkSources_1125905_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d33894fc-dca6-4d19-8307-afc8c1e287c5_isrForCrosstalkSources_1125905_22.stderr parsl.dataflow.dflow INFO: Task 1855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ec73759558def9c88c47879727faebe with result from task 1855 parsl.dataflow.dflow DEBUG: Task 1855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/0f86e093-b6a0-4eb8-9e9b-72e7f55eb9b6_isrForCrosstalkSources_1125920_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/0f86e093-b6a0-4eb8-9e9b-72e7f55eb9b6_isrForCrosstalkSources_1125920_53.stderr parsl.dataflow.dflow INFO: Task 1711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c8ba36c0d9a6040213ddaddc7815bfe with result from task 1711 parsl.dataflow.dflow DEBUG: Task 1711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8749d5bb-3c48-4ee3-b3ed-ff8dc872f740_isrForCrosstalkSources_1125944_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8749d5bb-3c48-4ee3-b3ed-ff8dc872f740_isrForCrosstalkSources_1125944_9.stderr parsl.dataflow.dflow INFO: Task 2067 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3c0b5356cc785c7de8299805ec29b75 with result from task 2067 parsl.dataflow.dflow DEBUG: Task 2067 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/5d534ab5-620a-4869-8774-485bb4dc6dd9_isrForCrosstalkSources_1125936_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/5d534ab5-620a-4869-8774-485bb4dc6dd9_isrForCrosstalkSources_1125936_30.stderr parsl.dataflow.dflow INFO: Task 2064 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1098c84fd918f4a233f6defb3b5efa5f with result from task 2064 parsl.dataflow.dflow DEBUG: Task 2064 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/ffcdeb66-ea74-45a0-b6c6-3317995683f4_isrForCrosstalkSources_1125904_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/ffcdeb66-ea74-45a0-b6c6-3317995683f4_isrForCrosstalkSources_1125904_42.stderr parsl.dataflow.dflow INFO: Task 1957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7cd9126c20cac1383b3763d12b5d889 with result from task 1957 parsl.dataflow.dflow DEBUG: Task 1957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/b0db1476-9cd7-45d8-92bc-84b4ef50217d_isrForCrosstalkSources_1125917_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/b0db1476-9cd7-45d8-92bc-84b4ef50217d_isrForCrosstalkSources_1125917_49.stderr parsl.dataflow.dflow INFO: Task 1989 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 905b5a5c5285e71bc62eee3d616cbab1 with result from task 1989 parsl.dataflow.dflow DEBUG: Task 1989 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/00b2e2ff-785b-4026-a8c5-7937d33d6f7d_isrForCrosstalkSources_1125927_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/00b2e2ff-785b-4026-a8c5-7937d33d6f7d_isrForCrosstalkSources_1125927_56.stderr parsl.dataflow.dflow INFO: Task 2004 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf8f10c44450c983441ed450d771be38 with result from task 2004 parsl.dataflow.dflow DEBUG: Task 2004 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/4864cb05-ebee-48da-9799-e1cbf939c9df_isrForCrosstalkSources_1125936_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/4864cb05-ebee-48da-9799-e1cbf939c9df_isrForCrosstalkSources_1125936_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:30 proc_lsst.multi:146 [INFO] found job 16896667 in provider checkpoint proc_lsst.multi INFO: found job 16896667 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2034 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b7a20f6da8b283d2e0de3b312b76708 with result from task 2034 parsl.dataflow.dflow DEBUG: Task 2034 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/fe1ca8a7-cbfa-4197-b277-99c7e1d74400_isrForCrosstalkSources_1125931_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/fe1ca8a7-cbfa-4197-b277-99c7e1d74400_isrForCrosstalkSources_1125931_41.stderr parsl.dataflow.dflow INFO: Task 1913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53e90528c0450b050cf08ebeab875171 with result from task 1913 parsl.dataflow.dflow DEBUG: Task 1913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/596d0218-9c50-4937-9c03-630939588634_isrForCrosstalkSources_1125913_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/596d0218-9c50-4937-9c03-630939588634_isrForCrosstalkSources_1125913_10.stderr parsl.dataflow.dflow INFO: Task 1985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d228f51e58610ff718871585ddfe1a7a with result from task 1985 parsl.dataflow.dflow DEBUG: Task 1985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/7d1ee071-be62-423d-8260-a7da86badea0_isrForCrosstalkSources_1125891_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/7d1ee071-be62-423d-8260-a7da86badea0_isrForCrosstalkSources_1125891_10.stderr parsl.dataflow.dflow INFO: Task 1939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f69ea10ec1e8badfebff0052315d2ce5 with result from task 1939 parsl.dataflow.dflow DEBUG: Task 1939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/82e0710c-1d43-491d-b32e-240db8460736_isrForCrosstalkSources_1125920_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/82e0710c-1d43-491d-b32e-240db8460736_isrForCrosstalkSources_1125920_23.stderr parsl.dataflow.dflow INFO: Task 1930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ba742a64f7a74fedd80a0d029a4d966 with result from task 1930 parsl.dataflow.dflow DEBUG: Task 1930 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/873dffcc-227c-443f-ae8c-8d3a9c940e1e_isrForCrosstalkSources_1125897_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/873dffcc-227c-443f-ae8c-8d3a9c940e1e_isrForCrosstalkSources_1125897_23.stderr parsl.dataflow.dflow INFO: Task 2020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e23e276fc7fcf81fbe430f083dc18819 with result from task 2020 parsl.dataflow.dflow DEBUG: Task 2020 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/f437ff3f-3e7b-4e5c-8fa3-76c87fe3fdc6_isrForCrosstalkSources_1125919_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/f437ff3f-3e7b-4e5c-8fa3-76c87fe3fdc6_isrForCrosstalkSources_1125919_38.stderr parsl.dataflow.dflow INFO: Task 2054 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bb4fedb3e11f703096e994240a41502 with result from task 2054 parsl.dataflow.dflow DEBUG: Task 2054 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/30ff5c45-c0e0-4ae4-990c-9f926c3659f9_isrForCrosstalkSources_1125901_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/30ff5c45-c0e0-4ae4-990c-9f926c3659f9_isrForCrosstalkSources_1125901_32.stderr parsl.dataflow.dflow INFO: Task 1976 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43896c27e2f5637c44cadaff9192acac with result from task 1976 parsl.dataflow.dflow DEBUG: Task 1976 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/eccecce3-b3d0-4ad2-ab45-64b4447b19c2_isrForCrosstalkSources_1125926_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/eccecce3-b3d0-4ad2-ab45-64b4447b19c2_isrForCrosstalkSources_1125926_44.stderr parsl.dataflow.dflow INFO: Task 1949 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc5f1b48c09cbdc2661de86dd6a330e2 with result from task 1949 parsl.dataflow.dflow DEBUG: Task 1949 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/c53d8bcc-ee5e-4b37-8a56-27029d9d1673_isrForCrosstalkSources_1125927_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/c53d8bcc-ee5e-4b37-8a56-27029d9d1673_isrForCrosstalkSources_1125927_27.stderr parsl.dataflow.dflow INFO: Task 1928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2b27d9bda7b102d2b809bc5780f2dd0 with result from task 1928 parsl.dataflow.dflow DEBUG: Task 1928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/2cba0780-ccab-43b3-a118-34621ee2328f_isrForCrosstalkSources_1125928_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/2cba0780-ccab-43b3-a118-34621ee2328f_isrForCrosstalkSources_1125928_51.stderr parsl.dataflow.dflow INFO: Task 1979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1d7e4c8a5429b87eec132d247d34251 with result from task 1979 parsl.dataflow.dflow DEBUG: Task 1979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/d9f8399f-7c48-49d1-a143-aa74a80681d6_isrForCrosstalkSources_1125903_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/d9f8399f-7c48-49d1-a143-aa74a80681d6_isrForCrosstalkSources_1125903_1.stderr parsl.dataflow.dflow INFO: Task 2006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8d0a168df31a637cbfc0072e6e55b41 with result from task 2006 parsl.dataflow.dflow DEBUG: Task 2006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/e53405a9-54f7-4a03-81fa-0dc677356073_isrForCrosstalkSources_1125936_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/e53405a9-54f7-4a03-81fa-0dc677356073_isrForCrosstalkSources_1125936_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:30 proc_lsst.multi:146 [INFO] found job 16896668 in provider checkpoint proc_lsst.multi INFO: found job 16896668 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 680a939e3d1c424b63541094ba5339b7 with result from task 2002 parsl.dataflow.dflow DEBUG: Task 2002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/b933cb4e-a443-406f-8073-a1299fab2d16_isrForCrosstalkSources_1125943_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/b933cb4e-a443-406f-8073-a1299fab2d16_isrForCrosstalkSources_1125943_61.stderr parsl.dataflow.dflow INFO: Task 2039 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb32c8aea45760488e9457799c5b275c with result from task 2039 parsl.dataflow.dflow DEBUG: Task 2039 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/437db6b0-a254-4410-8694-3742b9fcb078_isrForCrosstalkSources_1125935_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/437db6b0-a254-4410-8694-3742b9fcb078_isrForCrosstalkSources_1125935_41.stderr parsl.dataflow.dflow INFO: Task 1912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e62724708b1c8e252bb85cb087d136b with result from task 1912 parsl.dataflow.dflow DEBUG: Task 1912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/f746dcaa-59ac-444f-882f-af8ea6a2ed4d_isrForCrosstalkSources_1125934_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/f746dcaa-59ac-444f-882f-af8ea6a2ed4d_isrForCrosstalkSources_1125934_46.stderr parsl.dataflow.dflow INFO: Task 2013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef9724f340d635420cf6d48b8e3a1916 with result from task 2013 parsl.dataflow.dflow DEBUG: Task 2013 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/e7fd03bf-72d4-4238-8c43-f68799863ed9_isrForCrosstalkSources_1125896_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/e7fd03bf-72d4-4238-8c43-f68799863ed9_isrForCrosstalkSources_1125896_6.stderr parsl.dataflow.dflow INFO: Task 1952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63fe7cb42166d7f6756ffb9702d9e9ee with result from task 1952 parsl.dataflow.dflow DEBUG: Task 1952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/9ab75946-a69d-4efa-9187-3161324cf867_isrForCrosstalkSources_1125913_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/9ab75946-a69d-4efa-9187-3161324cf867_isrForCrosstalkSources_1125913_58.stderr parsl.dataflow.dflow INFO: Task 1948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01996b48a561cb1dfd17f5e8838e0b6e with result from task 1948 parsl.dataflow.dflow DEBUG: Task 1948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3f00b221-fb60-4f2b-a826-99284f7996f1_isrForCrosstalkSources_1125912_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3f00b221-fb60-4f2b-a826-99284f7996f1_isrForCrosstalkSources_1125912_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:31 proc_lsst.multi:146 [INFO] found job 16896669 in provider checkpoint proc_lsst.multi INFO: found job 16896669 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 635b4feb7ad8ee1256bb422cd03f3426 with result from task 1970 parsl.dataflow.dflow DEBUG: Task 1970 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/eaa8fdf6-58f3-4822-bae0-01e446110f36_isrForCrosstalkSources_1125930_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/eaa8fdf6-58f3-4822-bae0-01e446110f36_isrForCrosstalkSources_1125930_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:31 proc_lsst.multi:146 [INFO] found job 16896670 in provider checkpoint proc_lsst.multi INFO: found job 16896670 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ebc1f063f7a1f93bb1e575e9ab6d788 with result from task 1894 parsl.dataflow.dflow DEBUG: Task 1894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/74d90df9-1feb-4051-a2d0-263dc781ed2b_isrForCrosstalkSources_1125917_5.stdout parsl.dataflow.dflow INFO: Standard error for task 1894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/74d90df9-1feb-4051-a2d0-263dc781ed2b_isrForCrosstalkSources_1125917_5.stderr parsl.dataflow.dflow INFO: Task 2069 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 511358262deb8d5677d96a2c5a8e83ae with result from task 2069 parsl.dataflow.dflow DEBUG: Task 2069 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e68d00ab-cd59-42ec-9142-ff5e918cf99c_isrForCrosstalkSources_1125916_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e68d00ab-cd59-42ec-9142-ff5e918cf99c_isrForCrosstalkSources_1125916_27.stderr parsl.dataflow.dflow INFO: Task 2011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 114766df6370a24caf2a6a4b6287580d with result from task 2011 parsl.dataflow.dflow DEBUG: Task 2011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/71a9d350-1fbc-4cb2-a4fe-55c287ea4b24_isrForCrosstalkSources_1125893_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/71a9d350-1fbc-4cb2-a4fe-55c287ea4b24_isrForCrosstalkSources_1125893_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:31 proc_lsst.multi:146 [INFO] found job 16896671 in provider checkpoint proc_lsst.multi INFO: found job 16896671 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2035 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe842eb08f0bcc74c055b0fea8011868 with result from task 2035 parsl.dataflow.dflow DEBUG: Task 2035 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/17fe8818-a132-453d-a8f6-d84b33a3270a_isrForCrosstalkSources_1125893_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/17fe8818-a132-453d-a8f6-d84b33a3270a_isrForCrosstalkSources_1125893_4.stderr parsl.dataflow.dflow INFO: Task 1919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca7b83770a98ea6044f7e41cd37574f2 with result from task 1919 parsl.dataflow.dflow DEBUG: Task 1919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/56048bf6-8870-45bf-948f-e1e636100206_isrForCrosstalkSources_1125930_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/56048bf6-8870-45bf-948f-e1e636100206_isrForCrosstalkSources_1125930_37.stderr parsl.dataflow.dflow INFO: Task 2078 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e741851885d2c986d3d033df3f6887d3 with result from task 2078 parsl.dataflow.dflow DEBUG: Task 2078 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/27144643-844f-4240-b059-b7e6fa459f41_isrForCrosstalkSources_1125902_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/27144643-844f-4240-b059-b7e6fa459f41_isrForCrosstalkSources_1125902_13.stderr parsl.dataflow.dflow INFO: Task 1960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14179afabdd9c9d3fe5738b93a023238 with result from task 1960 parsl.dataflow.dflow DEBUG: Task 1960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/321173b2-d1fc-45b6-9a06-1ea86d2002af_isrForCrosstalkSources_1125946_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/321173b2-d1fc-45b6-9a06-1ea86d2002af_isrForCrosstalkSources_1125946_35.stderr parsl.dataflow.dflow INFO: Task 1958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a7d5a18574aa7f60e21b1b5e80ee780 with result from task 1958 parsl.dataflow.dflow DEBUG: Task 1958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/565bb661-b147-49c4-ad95-4529e3058231_isrForCrosstalkSources_1125943_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/565bb661-b147-49c4-ad95-4529e3058231_isrForCrosstalkSources_1125943_38.stderr parsl.dataflow.dflow INFO: Task 2058 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ad8941a16adf5ed3e7e758503b0938b with result from task 2058 parsl.dataflow.dflow DEBUG: Task 2058 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/12c1ce46-b3b0-492a-81f9-3fcb60b187d8_isrForCrosstalkSources_1125899_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/12c1ce46-b3b0-492a-81f9-3fcb60b187d8_isrForCrosstalkSources_1125899_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:31 proc_lsst.multi:146 [INFO] found job 16896672 in provider checkpoint proc_lsst.multi INFO: found job 16896672 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b53decaafbaa43f664d6d3b916c873a with result from task 1931 parsl.dataflow.dflow DEBUG: Task 1931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/ea8adfe4-34d8-4348-a351-8eb11c6395be_isrForCrosstalkSources_1125909_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/ea8adfe4-34d8-4348-a351-8eb11c6395be_isrForCrosstalkSources_1125909_30.stderr parsl.dataflow.dflow INFO: Task 1925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ae65b5757f069e83d980c9ad6554e67 with result from task 1925 parsl.dataflow.dflow DEBUG: Task 1925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/c9b99e80-0bdd-4ff6-a3c0-d7be8b4f5047_isrForCrosstalkSources_1125912_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/c9b99e80-0bdd-4ff6-a3c0-d7be8b4f5047_isrForCrosstalkSources_1125912_7.stderr parsl.dataflow.dflow INFO: Task 2041 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4224cbe41bdf9b63fee654baff4fa3c9 with result from task 2041 parsl.dataflow.dflow DEBUG: Task 2041 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e484790a-edf0-4bcc-a500-87cadbca1277_isrForCrosstalkSources_1125929_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e484790a-edf0-4bcc-a500-87cadbca1277_isrForCrosstalkSources_1125929_6.stderr parsl.dataflow.dflow INFO: Task 2038 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b531f310518489fc35ad9e4063d8bf20 with result from task 2038 parsl.dataflow.dflow DEBUG: Task 2038 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/ef4a13e2-b0a9-47e1-9092-7411d62671c1_isrForCrosstalkSources_1125920_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/ef4a13e2-b0a9-47e1-9092-7411d62671c1_isrForCrosstalkSources_1125920_4.stderr parsl.dataflow.dflow INFO: Task 1994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 459d9b2cd0911d9f28a2ee309a7274fb with result from task 1994 parsl.dataflow.dflow DEBUG: Task 1994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/27979002-f4a1-4985-947e-b59ac3268482_isrForCrosstalkSources_1125907_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/27979002-f4a1-4985-947e-b59ac3268482_isrForCrosstalkSources_1125907_36.stderr parsl.dataflow.dflow INFO: Task 2101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ae32254a9db4e7e035576c378023d6b with result from task 2101 parsl.dataflow.dflow DEBUG: Task 2101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/dc37727c-668a-48a2-9b25-3eeb3b714c31_isrForCrosstalkSources_1125927_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/dc37727c-668a-48a2-9b25-3eeb3b714c31_isrForCrosstalkSources_1125927_35.stderr parsl.dataflow.dflow INFO: Task 2053 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9f2e26106dc5b2cdb0895693e16dff9 with result from task 2053 parsl.dataflow.dflow DEBUG: Task 2053 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/e8cc4d04-7358-4568-aa15-460cb97ba5aa_isrForCrosstalkSources_1125947_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/e8cc4d04-7358-4568-aa15-460cb97ba5aa_isrForCrosstalkSources_1125947_46.stderr parsl.dataflow.dflow INFO: Task 1975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f120c42f21f2ed5548835ac789cf5c51 with result from task 1975 parsl.dataflow.dflow DEBUG: Task 1975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/156c9d77-c360-4ac9-84ce-6830e88b7a40_isrForCrosstalkSources_1125939_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/156c9d77-c360-4ac9-84ce-6830e88b7a40_isrForCrosstalkSources_1125939_11.stderr parsl.dataflow.dflow INFO: Task 2071 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6c3b4c754968dd3568bc11138547bee with result from task 2071 parsl.dataflow.dflow DEBUG: Task 2071 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/b50afede-3498-4fe6-9f78-215d95b971e3_isrForCrosstalkSources_1125900_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/b50afede-3498-4fe6-9f78-215d95b971e3_isrForCrosstalkSources_1125900_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:32 proc_lsst.multi:146 [INFO] found job 16896673 in provider checkpoint proc_lsst.multi INFO: found job 16896673 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2079 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15016aa83b95684b6f2d5fe73d258c56 with result from task 2079 parsl.dataflow.dflow DEBUG: Task 2079 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/a38893df-bb6d-4866-a971-0cfb6a964df4_isrForCrosstalkSources_1125920_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/a38893df-bb6d-4866-a971-0cfb6a964df4_isrForCrosstalkSources_1125920_20.stderr parsl.dataflow.dflow INFO: Task 2081 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e1c007716736ba3afe171161029823b with result from task 2081 parsl.dataflow.dflow DEBUG: Task 2081 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/ff0c05bc-e85d-4241-870b-37f834585f09_isrForCrosstalkSources_1125911_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/ff0c05bc-e85d-4241-870b-37f834585f09_isrForCrosstalkSources_1125911_39.stderr parsl.dataflow.dflow INFO: Task 1921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 669552bd460a642a345f80c311f9ff51 with result from task 1921 parsl.dataflow.dflow DEBUG: Task 1921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/0da4367c-a84f-4e6f-86a9-79f31954bc9c_isrForCrosstalkSources_1125923_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/0da4367c-a84f-4e6f-86a9-79f31954bc9c_isrForCrosstalkSources_1125923_50.stderr parsl.dataflow.dflow INFO: Task 2032 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66da70d2829ed3cb5a2c5d7b62b37daa with result from task 2032 parsl.dataflow.dflow DEBUG: Task 2032 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/e5fd10f4-aeb2-44b3-ad8d-4d0171346b25_isrForCrosstalkSources_1125924_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/e5fd10f4-aeb2-44b3-ad8d-4d0171346b25_isrForCrosstalkSources_1125924_46.stderr parsl.dataflow.dflow INFO: Task 1969 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcae32604eb00644edd9a2be4c078e4c with result from task 1969 parsl.dataflow.dflow DEBUG: Task 1969 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/7aaf0a1c-605b-4c1c-95b6-89ed69e8af83_isrForCrosstalkSources_1125913_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/7aaf0a1c-605b-4c1c-95b6-89ed69e8af83_isrForCrosstalkSources_1125913_44.stderr parsl.dataflow.dflow INFO: Task 2061 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15073426f2c48f392c87da3b450734a7 with result from task 2061 parsl.dataflow.dflow DEBUG: Task 2061 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/ac4599a7-61b9-421c-8268-2121bb7a2027_isrForCrosstalkSources_1125920_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/ac4599a7-61b9-421c-8268-2121bb7a2027_isrForCrosstalkSources_1125920_15.stderr parsl.dataflow.dflow INFO: Task 2103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cc4673875d884a1b47d0e5d86f98207 with result from task 2103 parsl.dataflow.dflow DEBUG: Task 2103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f6533b11-8eac-4829-b8d0-f8c5a554aff1_isrForCrosstalkSources_1125945_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f6533b11-8eac-4829-b8d0-f8c5a554aff1_isrForCrosstalkSources_1125945_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:32 proc_lsst.multi:146 [INFO] found job 16896674 in provider checkpoint proc_lsst.multi INFO: found job 16896674 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57dfde9f752e36503c8978a29bdc318a with result from task 1813 parsl.dataflow.dflow DEBUG: Task 1813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/7a159298-6a5b-46ca-8d41-2d25825f78b6_isrForCrosstalkSources_1125912_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/7a159298-6a5b-46ca-8d41-2d25825f78b6_isrForCrosstalkSources_1125912_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:32 proc_lsst.multi:146 [INFO] found job 16896675 in provider checkpoint proc_lsst.multi INFO: found job 16896675 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2057 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d128a0b16b82355efac50d2c4218e1c1 with result from task 2057 parsl.dataflow.dflow DEBUG: Task 2057 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/4e5dfb2f-2ae1-428c-8971-b2d5bb457cb9_isrForCrosstalkSources_1125894_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/4e5dfb2f-2ae1-428c-8971-b2d5bb457cb9_isrForCrosstalkSources_1125894_37.stderr parsl.dataflow.dflow INFO: Task 2070 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 316a02ce3bf7c01fae43c858296494aa with result from task 2070 parsl.dataflow.dflow DEBUG: Task 2070 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f31bc80f-9f8b-4ce5-8a17-84141dae09d6_isrForCrosstalkSources_1125939_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f31bc80f-9f8b-4ce5-8a17-84141dae09d6_isrForCrosstalkSources_1125939_23.stderr parsl.dataflow.dflow INFO: Task 1920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06d343609ce133aad0ef2788c7f068d5 with result from task 1920 parsl.dataflow.dflow DEBUG: Task 1920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/9d2589c6-b4f5-45dd-b695-43c407cac107_isrForCrosstalkSources_1125919_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/9d2589c6-b4f5-45dd-b695-43c407cac107_isrForCrosstalkSources_1125919_30.stderr parsl.dataflow.dflow INFO: Task 2059 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d4e81604e9ddc5098e7add28f58b7c3 with result from task 2059 parsl.dataflow.dflow DEBUG: Task 2059 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/e36fb971-4364-48e4-8494-df6db69469d5_isrForCrosstalkSources_1125891_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/e36fb971-4364-48e4-8494-df6db69469d5_isrForCrosstalkSources_1125891_22.stderr parsl.dataflow.dflow INFO: Task 1923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fc732709efab9961fb9889baa7fa363 with result from task 1923 parsl.dataflow.dflow DEBUG: Task 1923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/65107593-d678-4203-9793-a9f1e0331fe8_isrForCrosstalkSources_1125945_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/65107593-d678-4203-9793-a9f1e0331fe8_isrForCrosstalkSources_1125945_1.stderr parsl.dataflow.dflow INFO: Task 2051 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf569dc2eb6fbbf4d57beeab5355a490 with result from task 2051 parsl.dataflow.dflow DEBUG: Task 2051 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/c2579187-2861-44e0-b9d2-9b32b5efc9f0_isrForCrosstalkSources_1125903_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/c2579187-2861-44e0-b9d2-9b32b5efc9f0_isrForCrosstalkSources_1125903_40.stderr parsl.dataflow.dflow INFO: Task 2082 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc24f41574b3e74a09104d8b0e956c49 with result from task 2082 parsl.dataflow.dflow DEBUG: Task 2082 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/e0b67ef8-28d6-4c9e-a8c3-ffec87cc2c25_isrForCrosstalkSources_1125931_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/e0b67ef8-28d6-4c9e-a8c3-ffec87cc2c25_isrForCrosstalkSources_1125931_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:32 proc_lsst.multi:146 [INFO] found job 16896676 in provider checkpoint proc_lsst.multi INFO: found job 16896676 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 828fc0efde46f380d576e821cfced6f8 with result from task 1940 parsl.dataflow.dflow DEBUG: Task 1940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e85eff13-72cb-4d14-bc03-5a4825f94770_isrForCrosstalkSources_1125916_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e85eff13-72cb-4d14-bc03-5a4825f94770_isrForCrosstalkSources_1125916_3.stderr parsl.dataflow.dflow INFO: Task 2105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f70862fa386ab16f64de84a273c25a2e with result from task 2105 parsl.dataflow.dflow DEBUG: Task 2105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/aad7c8d3-1618-41be-b865-46bc4e73a360_isrForCrosstalkSources_1125931_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/aad7c8d3-1618-41be-b865-46bc4e73a360_isrForCrosstalkSources_1125931_22.stderr parsl.dataflow.dflow INFO: Task 1996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61340d28926ac14c6aec068149335515 with result from task 1996 parsl.dataflow.dflow DEBUG: Task 1996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/93fc0c6e-725c-4981-b407-0d5ff6ef4485_isrForCrosstalkSources_1125894_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/93fc0c6e-725c-4981-b407-0d5ff6ef4485_isrForCrosstalkSources_1125894_10.stderr parsl.dataflow.dflow INFO: Task 1965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41e0622cbb355b727ba236d0b7598c80 with result from task 1965 parsl.dataflow.dflow DEBUG: Task 1965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/c84baf7e-8e03-46ee-99c0-cd0c966318df_isrForCrosstalkSources_1125908_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/c84baf7e-8e03-46ee-99c0-cd0c966318df_isrForCrosstalkSources_1125908_44.stderr parsl.dataflow.dflow INFO: Task 2098 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed5bda66030db3df928e8c8cfbdc27cc with result from task 2098 parsl.dataflow.dflow DEBUG: Task 2098 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/498b6140-615f-4049-a389-c0e979304f13_isrForCrosstalkSources_1125907_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/498b6140-615f-4049-a389-c0e979304f13_isrForCrosstalkSources_1125907_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:33 proc_lsst.multi:146 [INFO] found job 16896678 in provider checkpoint proc_lsst.multi INFO: found job 16896678 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86e612321bf8eb3fd6279134b7549412 with result from task 1944 parsl.dataflow.dflow DEBUG: Task 1944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d4056072-718e-4c84-bbd4-23ad9d3fabfd_isrForCrosstalkSources_1125905_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/d4056072-718e-4c84-bbd4-23ad9d3fabfd_isrForCrosstalkSources_1125905_35.stderr parsl.dataflow.dflow INFO: Task 2080 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87b3bcc34f8e2e5b1571be07cfd8d2e8 with result from task 2080 parsl.dataflow.dflow DEBUG: Task 2080 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/19997bff-55e7-42bd-bf6b-9372c89f7b5f_isrForCrosstalkSources_1125913_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/19997bff-55e7-42bd-bf6b-9372c89f7b5f_isrForCrosstalkSources_1125913_29.stderr parsl.dataflow.dflow INFO: Task 2055 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f686dedae8a88b56db265cb82435c5f with result from task 2055 parsl.dataflow.dflow DEBUG: Task 2055 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/9bf7f4fa-8afb-4b3f-a6f7-c524602a1705_isrForCrosstalkSources_1125913_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/9bf7f4fa-8afb-4b3f-a6f7-c524602a1705_isrForCrosstalkSources_1125913_55.stderr parsl.dataflow.dflow INFO: Task 2025 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bed466d8a2ad60e86775591f5edd9f8 with result from task 2025 parsl.dataflow.dflow DEBUG: Task 2025 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/c37e41ab-6872-4ad1-975a-8db50e2dfa32_isrForCrosstalkSources_1125923_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/c37e41ab-6872-4ad1-975a-8db50e2dfa32_isrForCrosstalkSources_1125923_6.stderr parsl.dataflow.dflow INFO: Task 1909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d086e7e79558ac0c464cc4c54e8867c with result from task 1909 parsl.dataflow.dflow DEBUG: Task 1909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/9fb951ce-a124-4d53-bc06-c8839a5a453f_isrForCrosstalkSources_1125933_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/9fb951ce-a124-4d53-bc06-c8839a5a453f_isrForCrosstalkSources_1125933_11.stderr parsl.dataflow.dflow INFO: Task 1833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c34b84fe163d224325cf8528cb8abee7 with result from task 1833 parsl.dataflow.dflow DEBUG: Task 1833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/74f27a1f-fb27-4e34-ad70-7f7d10939d97_isrForCrosstalkSources_1125935_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/74f27a1f-fb27-4e34-ad70-7f7d10939d97_isrForCrosstalkSources_1125935_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:33 proc_lsst.multi:146 [INFO] found job 16896679 in provider checkpoint proc_lsst.multi INFO: found job 16896679 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2073 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 328c4ab219b2c53676971f41f77720bc with result from task 2073 parsl.dataflow.dflow DEBUG: Task 2073 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/bb335180-7aa1-439e-915e-e43e1327b425_isrForCrosstalkSources_1125889_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/bb335180-7aa1-439e-915e-e43e1327b425_isrForCrosstalkSources_1125889_22.stderr parsl.dataflow.dflow INFO: Task 2001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a706f4a51df9275e949865641206cf2d with result from task 2001 parsl.dataflow.dflow DEBUG: Task 2001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/89b7b11a-d135-486e-9758-182e6d77ad1f_isrForCrosstalkSources_1125914_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/89b7b11a-d135-486e-9758-182e6d77ad1f_isrForCrosstalkSources_1125914_7.stderr parsl.dataflow.dflow INFO: Task 2072 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0f025dfe76430a419812b907d6950c4 with result from task 2072 parsl.dataflow.dflow DEBUG: Task 2072 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/87e31c24-96b7-4819-826b-9f105fed9059_isrForCrosstalkSources_1125947_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/87e31c24-96b7-4819-826b-9f105fed9059_isrForCrosstalkSources_1125947_6.stderr parsl.dataflow.dflow INFO: Task 1968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a82cbb261c0de09143f03dd8cd285bf9 with result from task 1968 parsl.dataflow.dflow DEBUG: Task 1968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/0dec1cde-46a7-45b8-9f99-059a576462c0_isrForCrosstalkSources_1125893_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/0dec1cde-46a7-45b8-9f99-059a576462c0_isrForCrosstalkSources_1125893_62.stderr parsl.dataflow.dflow INFO: Task 2077 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcbea14622f7b5d5a1b94eaadbb345db with result from task 2077 parsl.dataflow.dflow DEBUG: Task 2077 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/86c06a0c-d8ca-4baa-bbc7-45f1c1d033d6_isrForCrosstalkSources_1125904_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/86c06a0c-d8ca-4baa-bbc7-45f1c1d033d6_isrForCrosstalkSources_1125904_44.stderr parsl.dataflow.dflow INFO: Task 2088 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c09794193b0b922896c2c67fa0a65f1 with result from task 2088 parsl.dataflow.dflow DEBUG: Task 2088 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/22fd47ef-9fdf-415e-9e72-e829d6176666_isrForCrosstalkSources_1125940_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/22fd47ef-9fdf-415e-9e72-e829d6176666_isrForCrosstalkSources_1125940_41.stderr parsl.dataflow.dflow INFO: Task 2128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23ed24e48900fcff0af82a088d4f1491 with result from task 2128 parsl.dataflow.dflow DEBUG: Task 2128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/7b8556e0-72ce-4dfd-8b6d-0301746a61af_isrForCrosstalkSources_1125921_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/7b8556e0-72ce-4dfd-8b6d-0301746a61af_isrForCrosstalkSources_1125921_16.stderr parsl.dataflow.dflow INFO: Task 2119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15cc913976d2e6697b122ec4aade325e with result from task 2119 parsl.dataflow.dflow DEBUG: Task 2119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/9dde083b-fd0a-418d-a432-2b5b3e5a650f_isrForCrosstalkSources_1125916_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/9dde083b-fd0a-418d-a432-2b5b3e5a650f_isrForCrosstalkSources_1125916_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:33 proc_lsst.multi:146 [INFO] found job 16896680 in provider checkpoint proc_lsst.multi INFO: found job 16896680 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2087 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46985df8e47078d9d41e3cea3464ab4b with result from task 2087 parsl.dataflow.dflow DEBUG: Task 2087 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/0e68c4bc-9814-4bba-b03c-ee98d11d7389_isrForCrosstalkSources_1125889_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/0e68c4bc-9814-4bba-b03c-ee98d11d7389_isrForCrosstalkSources_1125889_4.stderr parsl.dataflow.dflow INFO: Task 1953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 090f7948636cf2232f901857ced405e3 with result from task 1953 parsl.dataflow.dflow DEBUG: Task 1953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ca36cdeb-7e0d-4611-8733-b88c965c7fa6_isrForCrosstalkSources_1125902_48.stdout parsl.dataflow.dflow INFO: Standard error for task 1953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ca36cdeb-7e0d-4611-8733-b88c965c7fa6_isrForCrosstalkSources_1125902_48.stderr parsl.dataflow.dflow INFO: Task 1999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bae8b009984e3e22e9851db2cf21675a with result from task 1999 parsl.dataflow.dflow DEBUG: Task 1999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/f566a136-7762-459d-825b-c72cbf788394_isrForCrosstalkSources_1125902_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/f566a136-7762-459d-825b-c72cbf788394_isrForCrosstalkSources_1125902_57.stderr parsl.dataflow.dflow INFO: Task 2091 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 936df403ed9032f66025da7945f08423 with result from task 2091 parsl.dataflow.dflow DEBUG: Task 2091 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/332b64bf-655f-4b32-8eaa-9d1e2548b7c0_isrForCrosstalkSources_1125896_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/332b64bf-655f-4b32-8eaa-9d1e2548b7c0_isrForCrosstalkSources_1125896_48.stderr parsl.dataflow.dflow INFO: Task 2037 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8047fc17626f86918aa7c06561dd8e1d with result from task 2037 parsl.dataflow.dflow DEBUG: Task 2037 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/89774efb-5968-4f7c-930a-fb80d4d114fa_isrForCrosstalkSources_1125904_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/89774efb-5968-4f7c-930a-fb80d4d114fa_isrForCrosstalkSources_1125904_35.stderr parsl.dataflow.dflow INFO: Task 2118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e61896c443451425d679214b4d9515e with result from task 2118 parsl.dataflow.dflow DEBUG: Task 2118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/aa6247a2-0a8e-4a65-a608-11079638be48_isrForCrosstalkSources_1125902_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/aa6247a2-0a8e-4a65-a608-11079638be48_isrForCrosstalkSources_1125902_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:33 proc_lsst.multi:146 [INFO] found job 16896681 in provider checkpoint proc_lsst.multi INFO: found job 16896681 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73b017e69d49daf68969c3bbd3169519 with result from task 1897 parsl.dataflow.dflow DEBUG: Task 1897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/debb173a-38cc-48f3-bb47-f0e961158802_isrForCrosstalkSources_1125892_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/debb173a-38cc-48f3-bb47-f0e961158802_isrForCrosstalkSources_1125892_31.stderr parsl.dataflow.dflow INFO: Task 2062 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4c844b775f3b5ddd3e421991f738095 with result from task 2062 parsl.dataflow.dflow DEBUG: Task 2062 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/12d0ae75-bc60-44af-a081-e5deeda66bb6_isrForCrosstalkSources_1125935_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/12d0ae75-bc60-44af-a081-e5deeda66bb6_isrForCrosstalkSources_1125935_24.stderr parsl.dataflow.dflow INFO: Task 2045 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8df0a23890320de2da1b86240262975 with result from task 2045 parsl.dataflow.dflow DEBUG: Task 2045 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/f6caac42-fcef-4f20-b166-169aa52a77d7_isrForCrosstalkSources_1125915_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/f6caac42-fcef-4f20-b166-169aa52a77d7_isrForCrosstalkSources_1125915_23.stderr parsl.dataflow.dflow INFO: Task 1987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 675ab75493655d946d917dc076a6d64a with result from task 1987 parsl.dataflow.dflow DEBUG: Task 1987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/2e90a15f-56ee-4abe-bb89-f313d11cfab0_isrForCrosstalkSources_1125907_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/2e90a15f-56ee-4abe-bb89-f313d11cfab0_isrForCrosstalkSources_1125907_1.stderr parsl.dataflow.dflow INFO: Task 2065 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00b7ea6eeb11cd4bdac903cba8a3fa1a with result from task 2065 parsl.dataflow.dflow DEBUG: Task 2065 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/5eac6ba4-1997-4011-90ad-882489271096_isrForCrosstalkSources_1125905_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/5eac6ba4-1997-4011-90ad-882489271096_isrForCrosstalkSources_1125905_61.stderr parsl.dataflow.dflow INFO: Task 2115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d764e763083304cab2394b9becc8e07 with result from task 2115 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:34 proc_lsst.multi:146 [INFO] found job 16896682 in provider checkpoint proc_lsst.multi INFO: found job 16896682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/6b711895-faba-470e-b282-3786b9151907_isrForCrosstalkSources_1125937_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/6b711895-faba-470e-b282-3786b9151907_isrForCrosstalkSources_1125937_42.stderr parsl.dataflow.dflow INFO: Task 2068 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a7e70fa1354f077c2659e0b2b7a7e83 with result from task 2068 parsl.dataflow.dflow DEBUG: Task 2068 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/580ce628-f0aa-4520-935a-86a0515de81c_isrForCrosstalkSources_1125923_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/580ce628-f0aa-4520-935a-86a0515de81c_isrForCrosstalkSources_1125923_36.stderr parsl.dataflow.dflow INFO: Task 2052 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96bd9573bda854d210c908df25008a2c with result from task 2052 parsl.dataflow.dflow DEBUG: Task 2052 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/a728a166-3c23-44a2-9a11-23ae02196be9_isrForCrosstalkSources_1125907_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/a728a166-3c23-44a2-9a11-23ae02196be9_isrForCrosstalkSources_1125907_60.stderr parsl.dataflow.dflow INFO: Task 2097 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c240a42130ca532dbbfca380c66dbdc5 with result from task 2097 parsl.dataflow.dflow DEBUG: Task 2097 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/b088c8b8-7d5d-42ee-b575-18360994d9cf_isrForCrosstalkSources_1125940_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/b088c8b8-7d5d-42ee-b575-18360994d9cf_isrForCrosstalkSources_1125940_43.stderr parsl.dataflow.dflow INFO: Task 1978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbb41a7d7939ad793e20cf41403e4c73 with result from task 1978 parsl.dataflow.dflow DEBUG: Task 1978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/4b5d93dc-6959-40aa-bc93-34fc24478b2f_isrForCrosstalkSources_1125896_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/4b5d93dc-6959-40aa-bc93-34fc24478b2f_isrForCrosstalkSources_1125896_37.stderr parsl.dataflow.dflow INFO: Task 2040 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98217719b312c4c399839b146f6a49f8 with result from task 2040 parsl.dataflow.dflow DEBUG: Task 2040 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/bbbab053-3f7e-4e04-8b62-67c863a180bc_isrForCrosstalkSources_1125894_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/bbbab053-3f7e-4e04-8b62-67c863a180bc_isrForCrosstalkSources_1125894_52.stderr parsl.dataflow.dflow INFO: Task 1938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89a83d084141eca521c9e8472d63c086 with result from task 1938 parsl.dataflow.dflow DEBUG: Task 1938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/9bb03b0b-38b9-41d6-9426-2d8052590169_isrForCrosstalkSources_1125922_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/9bb03b0b-38b9-41d6-9426-2d8052590169_isrForCrosstalkSources_1125922_15.stderr parsl.dataflow.dflow INFO: Task 2027 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 064a2a21c3b042bbbee2593d67b3f27f with result from task 2027 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:34 proc_lsst.multi:146 [INFO] found job 16896683 in provider checkpoint proc_lsst.multi INFO: found job 16896683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2027 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0d109018-546a-4bca-9f3f-ac5f3f6c929b_isrForCrosstalkSources_1125904_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0d109018-546a-4bca-9f3f-ac5f3f6c929b_isrForCrosstalkSources_1125904_52.stderr parsl.dataflow.dflow INFO: Task 2047 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c36fd63f2073a3b7bd45a41a9193709 with result from task 2047 parsl.dataflow.dflow DEBUG: Task 2047 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/6664fc09-85d2-4e21-9ca0-549767597ff5_isrForCrosstalkSources_1125934_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/6664fc09-85d2-4e21-9ca0-549767597ff5_isrForCrosstalkSources_1125934_53.stderr parsl.dataflow.dflow INFO: Task 2130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72879d176cf17209febbeb6a2d8f6733 with result from task 2130 parsl.dataflow.dflow DEBUG: Task 2130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/7514619e-1e3d-47fe-a696-de63c81511aa_isrForCrosstalkSources_1125946_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/7514619e-1e3d-47fe-a696-de63c81511aa_isrForCrosstalkSources_1125946_27.stderr parsl.dataflow.dflow INFO: Task 1959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 619d5c4fdfa56fa0efb646cafa8ecdf1 with result from task 1959 parsl.dataflow.dflow DEBUG: Task 1959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/60c6de74-1f71-4765-b4a0-f796abe5752a_isrForCrosstalkSources_1125895_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/60c6de74-1f71-4765-b4a0-f796abe5752a_isrForCrosstalkSources_1125895_23.stderr parsl.dataflow.dflow INFO: Task 2117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42be4791077e58bcd1e3e3299e01a9e1 with result from task 2117 parsl.dataflow.dflow DEBUG: Task 2117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/e9b8d6b4-2476-4fb6-9c54-e2e81c5db3b0_isrForCrosstalkSources_1125915_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/e9b8d6b4-2476-4fb6-9c54-e2e81c5db3b0_isrForCrosstalkSources_1125915_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:34 proc_lsst.multi:146 [INFO] found job 16896684 in provider checkpoint proc_lsst.multi INFO: found job 16896684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccfe81a51b3419d1a32608ab988cd4d1 with result from task 1935 parsl.dataflow.dflow DEBUG: Task 1935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/d33bf2c8-e880-4eb2-ad51-f6ce024d7a78_isrForCrosstalkSources_1125937_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/d33bf2c8-e880-4eb2-ad51-f6ce024d7a78_isrForCrosstalkSources_1125937_37.stderr parsl.dataflow.dflow INFO: Task 2104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1271c627ff81c9997d09e1fc00e54e9c with result from task 2104 parsl.dataflow.dflow DEBUG: Task 2104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/7c4dbb21-a9e7-4208-a4c7-132ca730e11b_isrForCrosstalkSources_1125888_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/7c4dbb21-a9e7-4208-a4c7-132ca730e11b_isrForCrosstalkSources_1125888_14.stderr parsl.dataflow.dflow INFO: Task 2010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f77eb554ac474859533c552b2153d12f with result from task 2010 parsl.dataflow.dflow DEBUG: Task 2010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/8fc290c4-dcb3-47b1-b031-d9d04f9c9320_isrForCrosstalkSources_1125889_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/8fc290c4-dcb3-47b1-b031-d9d04f9c9320_isrForCrosstalkSources_1125889_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:34 proc_lsst.multi:146 [INFO] found job 16896685 in provider checkpoint proc_lsst.multi INFO: found job 16896685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65c45992e83d6b33bc0ffe090f4f1340 with result from task 1946 parsl.dataflow.dflow DEBUG: Task 1946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/1d61ec42-4d0b-49b9-8683-884e3a59e288_isrForCrosstalkSources_1125914_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/1d61ec42-4d0b-49b9-8683-884e3a59e288_isrForCrosstalkSources_1125914_47.stderr parsl.dataflow.dflow INFO: Task 1971 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5152f5ba6301904a189b4ecdfd5a128 with result from task 1971 parsl.dataflow.dflow DEBUG: Task 1971 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/06c9a8a4-6171-478d-8759-9be34671b41c_isrForCrosstalkSources_1125919_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/06c9a8a4-6171-478d-8759-9be34671b41c_isrForCrosstalkSources_1125919_9.stderr parsl.dataflow.dflow INFO: Task 2050 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21bfb4acb1a523720e4916c7cc95201a with result from task 2050 parsl.dataflow.dflow DEBUG: Task 2050 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d7a67265-743c-4d62-b9ae-a3d20ecc25da_isrForCrosstalkSources_1125890_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d7a67265-743c-4d62-b9ae-a3d20ecc25da_isrForCrosstalkSources_1125890_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:35 proc_lsst.multi:146 [INFO] found job 16896686 in provider checkpoint proc_lsst.multi INFO: found job 16896686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 1922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b793bcfae9e00f8a0e3ee9bcfc5e0ca with result from task 1922 parsl.dataflow.dflow DEBUG: Task 1922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ea5cce1a-080d-463e-ba90-e556b379fdeb_isrForCrosstalkSources_1125939_55.stdout parsl.dataflow.dflow INFO: Standard error for task 1922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ea5cce1a-080d-463e-ba90-e556b379fdeb_isrForCrosstalkSources_1125939_55.stderr parsl.dataflow.dflow INFO: Task 2003 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90cb72d9a15c7ab140d758a5fd4d2557 with result from task 2003 parsl.dataflow.dflow DEBUG: Task 2003 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/ca9dc70b-63e1-4466-8327-de3fae2ecd5b_isrForCrosstalkSources_1125907_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/ca9dc70b-63e1-4466-8327-de3fae2ecd5b_isrForCrosstalkSources_1125907_41.stderr parsl.dataflow.dflow INFO: Task 2066 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2387de0bf78e9e69cea1c09577e2bc60 with result from task 2066 parsl.dataflow.dflow DEBUG: Task 2066 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/13acd120-d30b-4708-87e9-87a8627af8f2_isrForCrosstalkSources_1125933_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/13acd120-d30b-4708-87e9-87a8627af8f2_isrForCrosstalkSources_1125933_55.stderr parsl.dataflow.dflow INFO: Task 2136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0dc51e1fc982f7a151bd2345ea2883e with result from task 2136 parsl.dataflow.dflow DEBUG: Task 2136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/4897e0a1-5d20-4aab-a72b-56785ba2fc56_isrForCrosstalkSources_1125914_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/4897e0a1-5d20-4aab-a72b-56785ba2fc56_isrForCrosstalkSources_1125914_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:35 proc_lsst.multi:146 [INFO] found job 16896687 in provider checkpoint proc_lsst.multi INFO: found job 16896687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0e8755b5a2a71cbf5f1bcda0414934c with result from task 2008 parsl.dataflow.dflow DEBUG: Task 2008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/a0ed7748-3a01-4704-a78f-a39203013884_isrForCrosstalkSources_1125898_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/a0ed7748-3a01-4704-a78f-a39203013884_isrForCrosstalkSources_1125898_53.stderr parsl.dataflow.dflow INFO: Task 2126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cc0360019a029951277135c02a1e7e6 with result from task 2126 parsl.dataflow.dflow DEBUG: Task 2126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/96f945a3-3dd9-45e4-9c29-79cbe4a6826d_isrForCrosstalkSources_1125895_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/96f945a3-3dd9-45e4-9c29-79cbe4a6826d_isrForCrosstalkSources_1125895_17.stderr parsl.dataflow.dflow INFO: Task 2093 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c08ee2040e47adf6852de726975bd53 with result from task 2093 parsl.dataflow.dflow DEBUG: Task 2093 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/138ebef2-cc56-453d-8d2e-017a63c12778_isrForCrosstalkSources_1125943_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/138ebef2-cc56-453d-8d2e-017a63c12778_isrForCrosstalkSources_1125943_60.stderr parsl.dataflow.dflow INFO: Task 2142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 509410072045a01a44e4839838af388e with result from task 2142 parsl.dataflow.dflow DEBUG: Task 2142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/47031b68-2ba9-4644-92f5-2218b6df5f9c_isrForCrosstalkSources_1125941_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/47031b68-2ba9-4644-92f5-2218b6df5f9c_isrForCrosstalkSources_1125941_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.dataflow.dflow INFO: Task 1850 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry bb03f102cded56735e235a9669496448 with result from task 1850 parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 1850 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:35 proc_lsst.multi:146 [INFO] found job 16896688 in provider checkpoint proc_lsst.multi INFO: found job 16896688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Standard output for task 1850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/ecf61a15-df88-46f5-8e5b-2bf913251787_isrForCrosstalkSources_1125908_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/ecf61a15-df88-46f5-8e5b-2bf913251787_isrForCrosstalkSources_1125908_21.stderr parsl.dataflow.dflow INFO: Task 2022 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 651d79f0fa2f334b13b73fc848716c8b with result from task 2022 parsl.dataflow.dflow DEBUG: Task 2022 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/973c9c40-2483-4386-bf1c-bbd8c7133490_isrForCrosstalkSources_1125901_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/973c9c40-2483-4386-bf1c-bbd8c7133490_isrForCrosstalkSources_1125901_30.stderr parsl.dataflow.dflow INFO: Task 2141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8daab1f84e8e4b99375ae9db0432761 with result from task 2141 parsl.dataflow.dflow DEBUG: Task 2141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/8bccd0e1-98d6-48f8-9041-0a95fb40e9ea_isrForCrosstalkSources_1125922_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/8bccd0e1-98d6-48f8-9041-0a95fb40e9ea_isrForCrosstalkSources_1125922_33.stderr parsl.dataflow.dflow INFO: Task 2076 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 992fd9197becb5621467d104fd9e0185 with result from task 2076 parsl.dataflow.dflow DEBUG: Task 2076 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/950851fc-5dcd-4c79-8953-99ea2be12d5c_isrForCrosstalkSources_1125941_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/950851fc-5dcd-4c79-8953-99ea2be12d5c_isrForCrosstalkSources_1125941_25.stderr parsl.dataflow.dflow INFO: Task 1918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d148c3281b1fe3b64736fb9262f11b63 with result from task 1918 parsl.dataflow.dflow DEBUG: Task 1918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/92a09669-7d38-40fb-ade1-7a88c0116812_isrForCrosstalkSources_1125938_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/92a09669-7d38-40fb-ade1-7a88c0116812_isrForCrosstalkSources_1125938_59.stderr parsl.dataflow.dflow INFO: Task 2044 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35e4fe4961dfa9edfffdf6fdd13c2a97 with result from task 2044 parsl.dataflow.dflow DEBUG: Task 2044 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/82fb2b76-f0d5-409f-80eb-2cebbd77cbfd_isrForCrosstalkSources_1125914_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/82fb2b76-f0d5-409f-80eb-2cebbd77cbfd_isrForCrosstalkSources_1125914_20.stderr parsl.dataflow.dflow INFO: Task 2033 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47f8399af60cb54e74daa789f4bb3aed with result from task 2033 parsl.dataflow.dflow DEBUG: Task 2033 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/75cede3c-12fe-4acb-9843-7178bfad1a82_isrForCrosstalkSources_1125910_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/75cede3c-12fe-4acb-9843-7178bfad1a82_isrForCrosstalkSources_1125910_48.stderr parsl.dataflow.dflow INFO: Task 2132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0507f5a68f57528b4d2171c20914f93 with result from task 2132 parsl.dataflow.dflow DEBUG: Task 2132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/7ac27915-cd73-4553-8686-b1b7ec296705_isrForCrosstalkSources_1125925_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/7ac27915-cd73-4553-8686-b1b7ec296705_isrForCrosstalkSources_1125925_7.stderr parsl.dataflow.dflow INFO: Task 2123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fd4cdff7bcd2de32d5d9705af34d417 with result from task 2123 parsl.dataflow.dflow DEBUG: Task 2123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/90a2fa78-6694-483f-94e4-794f33a12f91_isrForCrosstalkSources_1125945_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/90a2fa78-6694-483f-94e4-794f33a12f91_isrForCrosstalkSources_1125945_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:36 proc_lsst.multi:146 [INFO] found job 16896689 in provider checkpoint proc_lsst.multi INFO: found job 16896689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b21b148a1686caa9331f00d3f2a62b5 with result from task 2198 parsl.dataflow.dflow DEBUG: Task 2198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/6f9312c6-4c51-4c78-8fd8-fda45b494e7d_isrForCrosstalkSources_1125891_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/6f9312c6-4c51-4c78-8fd8-fda45b494e7d_isrForCrosstalkSources_1125891_46.stderr parsl.dataflow.dflow INFO: Task 2084 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3469f46cde1b954f11bec555bab13baf with result from task 2084 parsl.dataflow.dflow DEBUG: Task 2084 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/eb33f2a2-4fed-4ff6-98eb-116bb546b16b_isrForCrosstalkSources_1125892_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/eb33f2a2-4fed-4ff6-98eb-116bb546b16b_isrForCrosstalkSources_1125892_7.stderr parsl.dataflow.dflow INFO: Task 2075 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fc3da458866898211ed913b136e3a9e with result from task 2075 parsl.dataflow.dflow DEBUG: Task 2075 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/0e100123-3764-409e-945d-b4c987fd4c53_isrForCrosstalkSources_1125936_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/0e100123-3764-409e-945d-b4c987fd4c53_isrForCrosstalkSources_1125936_35.stderr parsl.dataflow.dflow INFO: Task 1974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52065301d3b86094a3fa51489fb6f7b9 with result from task 1974 parsl.dataflow.dflow DEBUG: Task 1974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/e48156e1-1440-41be-85cb-f6d056ed6259_isrForCrosstalkSources_1125935_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/e48156e1-1440-41be-85cb-f6d056ed6259_isrForCrosstalkSources_1125935_61.stderr parsl.dataflow.dflow INFO: Task 2133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be243659f5a39b36984ba0932b3ac3fd with result from task 2133 parsl.dataflow.dflow DEBUG: Task 2133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ba6cce75-d597-4e91-8748-f47f9d5f5d05_isrForCrosstalkSources_1125918_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ba6cce75-d597-4e91-8748-f47f9d5f5d05_isrForCrosstalkSources_1125918_39.stderr parsl.dataflow.dflow INFO: Task 2096 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2386ef8a8e70fdaf0f00f84ddc8e6fe with result from task 2096 parsl.dataflow.dflow DEBUG: Task 2096 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/e5091d49-0dd9-4d88-bdb0-2cc20a5d64f1_isrForCrosstalkSources_1125931_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/e5091d49-0dd9-4d88-bdb0-2cc20a5d64f1_isrForCrosstalkSources_1125931_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:36 proc_lsst.multi:146 [INFO] found job 16896690 in provider checkpoint proc_lsst.multi INFO: found job 16896690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1c546edbd485d803bede964a000a5ec with result from task 2000 parsl.dataflow.dflow DEBUG: Task 2000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/2429dfc1-5725-4df9-9059-97bf883f20e9_isrForCrosstalkSources_1125899_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/2429dfc1-5725-4df9-9059-97bf883f20e9_isrForCrosstalkSources_1125899_37.stderr parsl.dataflow.dflow INFO: Task 2056 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a1da0638bc085e9a99fab847b88e030 with result from task 2056 parsl.dataflow.dflow DEBUG: Task 2056 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/d7067b4b-d0ba-4c0c-bb7b-a8c49636f094_isrForCrosstalkSources_1125900_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/d7067b4b-d0ba-4c0c-bb7b-a8c49636f094_isrForCrosstalkSources_1125900_52.stderr parsl.dataflow.dflow INFO: Task 2063 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb414f57a75210a96f8fb58a6a0d8a21 with result from task 2063 parsl.dataflow.dflow DEBUG: Task 2063 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/cf54d74a-cadc-4043-9ffe-27801c828ee7_isrForCrosstalkSources_1125944_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/cf54d74a-cadc-4043-9ffe-27801c828ee7_isrForCrosstalkSources_1125944_49.stderr parsl.dataflow.dflow INFO: Task 2090 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0259f4d2e2febad6bc8fe9f21fee3d23 with result from task 2090 parsl.dataflow.dflow DEBUG: Task 2090 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/abe69bc9-85a3-44ed-89b2-551135d909ca_isrForCrosstalkSources_1125889_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/abe69bc9-85a3-44ed-89b2-551135d909ca_isrForCrosstalkSources_1125889_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:36 proc_lsst.multi:146 [INFO] found job 16896691 in provider checkpoint proc_lsst.multi INFO: found job 16896691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 491899dcccd2b1e9c96d8f9671016d4f with result from task 2212 parsl.dataflow.dflow DEBUG: Task 2212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/27637ab8-70fe-4286-b636-2e8297ecde66_isrForCrosstalkSources_1125922_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/27637ab8-70fe-4286-b636-2e8297ecde66_isrForCrosstalkSources_1125922_26.stderr parsl.dataflow.dflow INFO: Task 2108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fedb92454319bebdda70a10934609ec with result from task 2108 parsl.dataflow.dflow DEBUG: Task 2108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/5d8123f3-e5aa-4d24-9994-c661a433884c_isrForCrosstalkSources_1125898_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/5d8123f3-e5aa-4d24-9994-c661a433884c_isrForCrosstalkSources_1125898_50.stderr parsl.dataflow.dflow INFO: Task 2144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00e923b770db972c1cfba70fc2c44852 with result from task 2144 parsl.dataflow.dflow DEBUG: Task 2144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/7a2cc30f-b9e5-49da-bd8b-43f7063482dc_isrForCrosstalkSources_1125945_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/7a2cc30f-b9e5-49da-bd8b-43f7063482dc_isrForCrosstalkSources_1125945_58.stderr parsl.dataflow.dflow INFO: Task 2129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 135d2275dfbab1c31c5cf9282e083f46 with result from task 2129 parsl.dataflow.dflow DEBUG: Task 2129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/0dffbb12-26b9-4ffc-bad5-aac369baf1d2_isrForCrosstalkSources_1125925_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/0dffbb12-26b9-4ffc-bad5-aac369baf1d2_isrForCrosstalkSources_1125925_17.stderr parsl.dataflow.dflow INFO: Task 2222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bdaa777cd3037ac330b7ad02a83ed51 with result from task 2222 parsl.dataflow.dflow DEBUG: Task 2222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/a232ca2b-5551-45fa-81f4-6f3dc2f782be_isrForCrosstalkSources_1125947_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/a232ca2b-5551-45fa-81f4-6f3dc2f782be_isrForCrosstalkSources_1125947_51.stderr parsl.dataflow.dflow INFO: Task 2120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06ae4099921a9aa89612928d47d6e435 with result from task 2120 parsl.dataflow.dflow DEBUG: Task 2120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/8b7d6ae0-825a-4615-9716-9fc04bf860d6_isrForCrosstalkSources_1125906_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/8b7d6ae0-825a-4615-9716-9fc04bf860d6_isrForCrosstalkSources_1125906_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:36 proc_lsst.multi:146 [INFO] found job 16896692 in provider checkpoint proc_lsst.multi INFO: found job 16896692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 392baefeb2e6c3f45beb816d7448d453 with result from task 2205 parsl.dataflow.dflow DEBUG: Task 2205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/445000a4-8d59-4063-b12a-90c41fcf8f59_isrForCrosstalkSources_1125897_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/445000a4-8d59-4063-b12a-90c41fcf8f59_isrForCrosstalkSources_1125897_39.stderr parsl.dataflow.dflow INFO: Task 2162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 834bf622ae8efb89755e7d77acb99f7c with result from task 2162 parsl.dataflow.dflow DEBUG: Task 2162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/693ebe42-ed5f-4c78-b5e4-42d509d2ee1e_isrForCrosstalkSources_1125918_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/693ebe42-ed5f-4c78-b5e4-42d509d2ee1e_isrForCrosstalkSources_1125918_19.stderr parsl.dataflow.dflow INFO: Task 2042 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fdb3d01643d2d1515dd6c390301cb35 with result from task 2042 parsl.dataflow.dflow DEBUG: Task 2042 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/16e7aebc-d410-4076-b5e6-999499382d5a_isrForCrosstalkSources_1125933_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/16e7aebc-d410-4076-b5e6-999499382d5a_isrForCrosstalkSources_1125933_21.stderr parsl.dataflow.dflow INFO: Task 2148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7a22822c09a0207a0137b9f5ed51db5 with result from task 2148 parsl.dataflow.dflow DEBUG: Task 2148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/ba81997b-23d1-4e90-8f12-4aa892d33d2a_isrForCrosstalkSources_1125921_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/ba81997b-23d1-4e90-8f12-4aa892d33d2a_isrForCrosstalkSources_1125921_24.stderr parsl.dataflow.dflow INFO: Task 1977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39c9960fddddcdccdab668075e1fc7a8 with result from task 1977 parsl.dataflow.dflow DEBUG: Task 1977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/310f073a-ea81-4f4c-8f52-53cbcef5b626_isrForCrosstalkSources_1125945_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/310f073a-ea81-4f4c-8f52-53cbcef5b626_isrForCrosstalkSources_1125945_54.stderr parsl.dataflow.dflow INFO: Task 2186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8842813924cb873c97e71f168afe0115 with result from task 2186 parsl.dataflow.dflow DEBUG: Task 2186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/9ee939d3-66d9-4a85-b3e6-e61534285df0_isrForCrosstalkSources_1125939_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/9ee939d3-66d9-4a85-b3e6-e61534285df0_isrForCrosstalkSources_1125939_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:37 proc_lsst.multi:146 [INFO] found job 16896693 in provider checkpoint proc_lsst.multi INFO: found job 16896693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8eabcc6eaa95d32bd7b40a2187be61a2 with result from task 2216 parsl.dataflow.dflow DEBUG: Task 2216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/c3aae125-0913-4a35-9f46-09377b2320d1_isrForCrosstalkSources_1125896_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/c3aae125-0913-4a35-9f46-09377b2320d1_isrForCrosstalkSources_1125896_7.stderr parsl.dataflow.dflow INFO: Task 2127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 223ed84c2fc9509fd7eb6e73b859d9f8 with result from task 2127 parsl.dataflow.dflow DEBUG: Task 2127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/ca58a9d5-9063-452f-8bb5-5669f6a2a809_isrForCrosstalkSources_1125909_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/ca58a9d5-9063-452f-8bb5-5669f6a2a809_isrForCrosstalkSources_1125909_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:37 proc_lsst.multi:146 [INFO] found job 16896694 in provider checkpoint proc_lsst.multi INFO: found job 16896694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58c4466b28651413cc558932bac941b2 with result from task 2150 parsl.dataflow.dflow DEBUG: Task 2150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/1c3da744-13ec-4996-b823-e1a0c7ea4527_isrForCrosstalkSources_1125917_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/1c3da744-13ec-4996-b823-e1a0c7ea4527_isrForCrosstalkSources_1125917_46.stderr parsl.dataflow.dflow INFO: Task 2146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ced3be8610897cbd77fd1bc65fc59cdb with result from task 2146 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:37 proc_lsst.multi:146 [INFO] found job 16896695 in provider checkpoint proc_lsst.multi INFO: found job 16896695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/15086f69-5bb4-4cbc-b706-1a3b44788bf0_isrForCrosstalkSources_1125916_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/15086f69-5bb4-4cbc-b706-1a3b44788bf0_isrForCrosstalkSources_1125916_5.stderr parsl.dataflow.dflow INFO: Task 2179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec1dbec8d038ec40316d9d0bded8429b with result from task 2179 parsl.dataflow.dflow DEBUG: Task 2179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/347f5431-a5c7-4c2d-a0e5-42bd343950bb_isrForCrosstalkSources_1125893_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/347f5431-a5c7-4c2d-a0e5-42bd343950bb_isrForCrosstalkSources_1125893_22.stderr parsl.dataflow.dflow INFO: Task 2110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e93a7c90f70b51e471da4d2020a8bf73 with result from task 2110 parsl.dataflow.dflow DEBUG: Task 2110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/49792d21-c2fe-47fa-a401-30623a09f1a2_isrForCrosstalkSources_1125941_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/49792d21-c2fe-47fa-a401-30623a09f1a2_isrForCrosstalkSources_1125941_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:37 proc_lsst.multi:146 [INFO] found job 16896696 in provider checkpoint proc_lsst.multi INFO: found job 16896696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39110028ebcb7a7a49c6ff69b38db373 with result from task 2152 parsl.dataflow.dflow DEBUG: Task 2152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/916ab6cf-3e79-4c5b-a833-4723159a2ee4_isrForCrosstalkSources_1125921_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/916ab6cf-3e79-4c5b-a833-4723159a2ee4_isrForCrosstalkSources_1125921_29.stderr parsl.dataflow.dflow INFO: Task 2154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fe19d2f79bb8e2dac0ca0c3e9f0ee13 with result from task 2154 parsl.dataflow.dflow DEBUG: Task 2154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/a79d0588-6250-42cd-b798-085a96e8bb4f_isrForCrosstalkSources_1125895_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/a79d0588-6250-42cd-b798-085a96e8bb4f_isrForCrosstalkSources_1125895_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:38 proc_lsst.multi:146 [INFO] found job 16896697 in provider checkpoint proc_lsst.multi INFO: found job 16896697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c847a7f6ecb7007e9b61ad76fe3ecb2f with result from task 2158 parsl.dataflow.dflow DEBUG: Task 2158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/e4719521-e0ec-4a77-9fac-d94865793df7_isrForCrosstalkSources_1125919_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/e4719521-e0ec-4a77-9fac-d94865793df7_isrForCrosstalkSources_1125919_16.stderr parsl.dataflow.dflow INFO: Task 2145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry add9d8f5da6dda60d1a877045009f662 with result from task 2145 parsl.dataflow.dflow DEBUG: Task 2145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/0e8230c4-1d49-4ecd-abcc-3bcbffb7bb61_isrForCrosstalkSources_1125942_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/0e8230c4-1d49-4ecd-abcc-3bcbffb7bb61_isrForCrosstalkSources_1125942_6.stderr parsl.dataflow.dflow INFO: Task 2171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f26480bd4be6f72a572c4d8211d97673 with result from task 2171 parsl.dataflow.dflow DEBUG: Task 2171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/f6fe3b42-1f94-4f06-9267-7964a7a6b9a9_isrForCrosstalkSources_1125933_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/f6fe3b42-1f94-4f06-9267-7964a7a6b9a9_isrForCrosstalkSources_1125933_20.stderr parsl.dataflow.dflow INFO: Task 2246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0130f063338580f5ef97c3697466b4d with result from task 2246 parsl.dataflow.dflow DEBUG: Task 2246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/e31a579e-b9ef-4da3-986b-a04cd2a16599_isrForCrosstalkSources_1125923_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/e31a579e-b9ef-4da3-986b-a04cd2a16599_isrForCrosstalkSources_1125923_30.stderr parsl.dataflow.dflow INFO: Task 2138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3cd305c5406311317dfecb5fa1d8c67 with result from task 2138 parsl.dataflow.dflow DEBUG: Task 2138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/49f0a26b-293c-4aa4-ab1b-f5d0533e5c94_isrForCrosstalkSources_1125913_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/49f0a26b-293c-4aa4-ab1b-f5d0533e5c94_isrForCrosstalkSources_1125913_59.stderr parsl.dataflow.dflow INFO: Task 2204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fbf7d63555bc8834483a9a36a5a4f76 with result from task 2204 parsl.dataflow.dflow DEBUG: Task 2204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/4c2f2fa4-b709-4f32-9621-4eb481d278df_isrForCrosstalkSources_1125894_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/4c2f2fa4-b709-4f32-9621-4eb481d278df_isrForCrosstalkSources_1125894_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:38 proc_lsst.multi:146 [INFO] found job 16896698 in provider checkpoint proc_lsst.multi INFO: found job 16896698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9327f3a6b8cb4769f94de0fecd525fb6 with result from task 2149 parsl.dataflow.dflow DEBUG: Task 2149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/4190716a-b3aa-41ab-993f-090bd052a472_isrForCrosstalkSources_1125892_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/4190716a-b3aa-41ab-993f-090bd052a472_isrForCrosstalkSources_1125892_41.stderr parsl.dataflow.dflow INFO: Task 2137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffae66cd978c7dd0af4ad5a43bf023a6 with result from task 2137 parsl.dataflow.dflow DEBUG: Task 2137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/1f63d591-24a3-4181-90b8-f9ed92cb25b6_isrForCrosstalkSources_1125914_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/1f63d591-24a3-4181-90b8-f9ed92cb25b6_isrForCrosstalkSources_1125914_6.stderr parsl.dataflow.dflow INFO: Task 2121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 365c41311ba72f88ea3a69c9846ad62c with result from task 2121 parsl.dataflow.dflow DEBUG: Task 2121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/11d9dfd2-8aa0-4a4c-bcdd-e6a8e1957f41_isrForCrosstalkSources_1125910_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/11d9dfd2-8aa0-4a4c-bcdd-e6a8e1957f41_isrForCrosstalkSources_1125910_52.stderr parsl.dataflow.dflow INFO: Task 2166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4305b57940a528ff244b057ed682b1a8 with result from task 2166 parsl.dataflow.dflow DEBUG: Task 2166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/a1de2e30-9371-4955-b9a3-90147d300734_isrForCrosstalkSources_1125914_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/a1de2e30-9371-4955-b9a3-90147d300734_isrForCrosstalkSources_1125914_1.stderr parsl.dataflow.dflow INFO: Task 2156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53f660bbd0677747d6d3d5555b823606 with result from task 2156 parsl.dataflow.dflow DEBUG: Task 2156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/2676f32f-b2db-40cf-a312-3df0cac28b64_isrForCrosstalkSources_1125947_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/2676f32f-b2db-40cf-a312-3df0cac28b64_isrForCrosstalkSources_1125947_56.stderr parsl.dataflow.dflow INFO: Task 2028 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 105e4b1b72a81b7d8ab411a84ac65331 with result from task 2028 parsl.dataflow.dflow DEBUG: Task 2028 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/88b435c7-04a3-4508-9859-cfaa07b0688e_isrForCrosstalkSources_1125892_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/88b435c7-04a3-4508-9859-cfaa07b0688e_isrForCrosstalkSources_1125892_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:38 proc_lsst.multi:146 [INFO] found job 16896699 in provider checkpoint proc_lsst.multi INFO: found job 16896699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9627c9bd802422c2623dc0d75d3992e3 with result from task 2140 parsl.dataflow.dflow DEBUG: Task 2140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/d8efcd71-3b76-43bb-bc98-ca905ecd33bb_isrForCrosstalkSources_1125928_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/d8efcd71-3b76-43bb-bc98-ca905ecd33bb_isrForCrosstalkSources_1125928_55.stderr parsl.dataflow.dflow INFO: Task 2243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55199da927d7846386751ce3d49e72d0 with result from task 2243 parsl.dataflow.dflow DEBUG: Task 2243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/22b4f0f4-8351-41b0-9f89-8288d8c4526e_isrForCrosstalkSources_1125922_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/22b4f0f4-8351-41b0-9f89-8288d8c4526e_isrForCrosstalkSources_1125922_39.stderr parsl.dataflow.dflow INFO: Task 2241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2af11108fc43849f0eaadd7962807f2 with result from task 2241 parsl.dataflow.dflow DEBUG: Task 2241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/ffa94ab9-141a-4a4e-b064-809262f5c02d_isrForCrosstalkSources_1125888_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/ffa94ab9-141a-4a4e-b064-809262f5c02d_isrForCrosstalkSources_1125888_12.stderr parsl.dataflow.dflow INFO: Task 2143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f5ce6118f6f4e1e9a52420281619533 with result from task 2143 parsl.dataflow.dflow DEBUG: Task 2143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/4ca6c849-7a97-490e-876c-52b1f901e94b_isrForCrosstalkSources_1125922_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/4ca6c849-7a97-490e-876c-52b1f901e94b_isrForCrosstalkSources_1125922_52.stderr parsl.dataflow.dflow INFO: Task 2226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9d8429f08298ac23815360d8d8c0eff with result from task 2226 parsl.dataflow.dflow DEBUG: Task 2226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/7e7e039f-b978-4d0f-8fc9-a8626bed8a24_isrForCrosstalkSources_1125890_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/7e7e039f-b978-4d0f-8fc9-a8626bed8a24_isrForCrosstalkSources_1125890_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:38 proc_lsst.multi:146 [INFO] found job 16896700 in provider checkpoint proc_lsst.multi INFO: found job 16896700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd10780e80900825a815b4e3f9e49a53 with result from task 2215 parsl.dataflow.dflow DEBUG: Task 2215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/0baa5af4-0862-41b3-a171-af35fa38e50f_isrForCrosstalkSources_1125939_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/0baa5af4-0862-41b3-a171-af35fa38e50f_isrForCrosstalkSources_1125939_9.stderr parsl.dataflow.dflow INFO: Task 2202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a24b9797b12b9918773e8e4cc3bebb89 with result from task 2202 parsl.dataflow.dflow DEBUG: Task 2202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/1799fa02-93d3-4c38-baf9-29cec97490ad_isrForCrosstalkSources_1125905_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/1799fa02-93d3-4c38-baf9-29cec97490ad_isrForCrosstalkSources_1125905_27.stderr parsl.dataflow.dflow INFO: Task 2188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 523ba4bba64264a1cacaa207573d1ba3 with result from task 2188 parsl.dataflow.dflow DEBUG: Task 2188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/237ef975-8eee-4b5b-9879-283a1a6f9763_isrForCrosstalkSources_1125947_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/237ef975-8eee-4b5b-9879-283a1a6f9763_isrForCrosstalkSources_1125947_29.stderr parsl.dataflow.dflow INFO: Task 2182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a8e20399a6f606acaa8cf5a8a8d3cce with result from task 2182 parsl.dataflow.dflow DEBUG: Task 2182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/a61c75bb-7af4-4ed7-800d-f955251fbb58_isrForCrosstalkSources_1125947_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/a61c75bb-7af4-4ed7-800d-f955251fbb58_isrForCrosstalkSources_1125947_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:38 proc_lsst.multi:146 [INFO] found job 16896701 in provider checkpoint proc_lsst.multi INFO: found job 16896701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b2aeebe46aaa829d84e7bef21c98102 with result from task 2196 parsl.dataflow.dflow DEBUG: Task 2196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/7f40f5d0-5cab-428e-9290-5e1d46e17dc6_isrForCrosstalkSources_1125939_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/7f40f5d0-5cab-428e-9290-5e1d46e17dc6_isrForCrosstalkSources_1125939_2.stderr parsl.dataflow.dflow INFO: Task 2147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 737a6afe8853d4bea092f6f2295bef3b with result from task 2147 parsl.dataflow.dflow DEBUG: Task 2147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/c6c291b5-b192-4d82-9852-d59f276c8596_isrForCrosstalkSources_1125937_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/c6c291b5-b192-4d82-9852-d59f276c8596_isrForCrosstalkSources_1125937_48.stderr parsl.dataflow.dflow INFO: Task 2191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b797d33edc78578ab13fb6c4863842e9 with result from task 2191 parsl.dataflow.dflow DEBUG: Task 2191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/d6eb359c-6c81-4f96-a914-2719b5af1560_isrForCrosstalkSources_1125921_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/d6eb359c-6c81-4f96-a914-2719b5af1560_isrForCrosstalkSources_1125921_34.stderr parsl.dataflow.dflow INFO: Task 2192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a5a94bdd9126ea6b8c9e60d4408d3d3 with result from task 2192 parsl.dataflow.dflow DEBUG: Task 2192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/9e9e2e35-67a1-4b1d-8133-1456af911938_isrForCrosstalkSources_1125905_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/9e9e2e35-67a1-4b1d-8133-1456af911938_isrForCrosstalkSources_1125905_15.stderr parsl.dataflow.dflow INFO: Task 2218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee99af57fd909aa631df88f2bd2cfc14 with result from task 2218 parsl.dataflow.dflow DEBUG: Task 2218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/f7560a04-07c3-4878-8da5-6a03a0856eaf_isrForCrosstalkSources_1125898_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/f7560a04-07c3-4878-8da5-6a03a0856eaf_isrForCrosstalkSources_1125898_45.stderr parsl.dataflow.dflow INFO: Task 2195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99fac64849f1ff6d9313a0f29c5fc06e with result from task 2195 parsl.dataflow.dflow DEBUG: Task 2195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/35cbc9b3-f0bd-449a-a847-d3bd03d8ad77_isrForCrosstalkSources_1125940_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/35cbc9b3-f0bd-449a-a847-d3bd03d8ad77_isrForCrosstalkSources_1125940_20.stderr parsl.dataflow.dflow INFO: Task 2227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30175459c87bd790d40dfdbc6e868aad with result from task 2227 parsl.dataflow.dflow DEBUG: Task 2227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/b2517d6c-9f4d-4131-9b0b-47825a76290b_isrForCrosstalkSources_1125942_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/b2517d6c-9f4d-4131-9b0b-47825a76290b_isrForCrosstalkSources_1125942_37.stderr parsl.dataflow.dflow INFO: Task 2251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1011b59d34bb5588ed4948f43e13832 with result from task 2251 parsl.dataflow.dflow DEBUG: Task 2251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/3b64d45b-4c56-4c25-a20c-0220ba47f1bc_isrForCrosstalkSources_1125935_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/3b64d45b-4c56-4c25-a20c-0220ba47f1bc_isrForCrosstalkSources_1125935_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:39 proc_lsst.multi:146 [INFO] found job 16896702 in provider checkpoint proc_lsst.multi INFO: found job 16896702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b525fe7488b81f9f287fd03995cfd32 with result from task 2228 parsl.dataflow.dflow DEBUG: Task 2228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/fead0822-7f5c-447e-8ddb-02aea6300e91_isrForCrosstalkSources_1125906_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/fead0822-7f5c-447e-8ddb-02aea6300e91_isrForCrosstalkSources_1125906_13.stderr parsl.dataflow.dflow INFO: Task 2135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a1eedd9e6cfdc4343cf289a38b5872e with result from task 2135 parsl.dataflow.dflow DEBUG: Task 2135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/46b6c499-1fe2-4e50-90a1-547da70dff2f_isrForCrosstalkSources_1125906_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/46b6c499-1fe2-4e50-90a1-547da70dff2f_isrForCrosstalkSources_1125906_58.stderr parsl.dataflow.dflow INFO: Task 2230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81c40cf1573443d42a531bbb073fd4e2 with result from task 2230 parsl.dataflow.dflow DEBUG: Task 2230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/71db5e86-189b-4ef2-a181-a8b3ccd89b16_isrForCrosstalkSources_1125945_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/71db5e86-189b-4ef2-a181-a8b3ccd89b16_isrForCrosstalkSources_1125945_57.stderr parsl.dataflow.dflow INFO: Task 2164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 919378ab5094a14acc3e19fc001d1a0d with result from task 2164 parsl.dataflow.dflow DEBUG: Task 2164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/65aeabb2-e8ca-4362-975a-1f0a09a516d7_isrForCrosstalkSources_1125941_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/65aeabb2-e8ca-4362-975a-1f0a09a516d7_isrForCrosstalkSources_1125941_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:39 proc_lsst.multi:146 [INFO] found job 16896703 in provider checkpoint proc_lsst.multi INFO: found job 16896703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2099 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81eba4f68601a1cee48496df2214b001 with result from task 2099 parsl.dataflow.dflow DEBUG: Task 2099 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/71d24a3b-8f7c-4015-9c31-8e153dac30aa_isrForCrosstalkSources_1125925_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/71d24a3b-8f7c-4015-9c31-8e153dac30aa_isrForCrosstalkSources_1125925_31.stderr parsl.dataflow.dflow INFO: Task 2181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad3d2b70246fcc4eb164df6ef123142b with result from task 2181 parsl.dataflow.dflow DEBUG: Task 2181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/0bae3cce-88ed-47f5-b566-1ff8240ef289_isrForCrosstalkSources_1125917_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/0bae3cce-88ed-47f5-b566-1ff8240ef289_isrForCrosstalkSources_1125917_17.stderr parsl.dataflow.dflow INFO: Task 2236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3554ccea9277cbb61f810da8cbd78f5 with result from task 2236 parsl.dataflow.dflow DEBUG: Task 2236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/af867f5c-57b1-43d7-abd1-17247b0933ec_isrForCrosstalkSources_1125915_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/af867f5c-57b1-43d7-abd1-17247b0933ec_isrForCrosstalkSources_1125915_19.stderr parsl.dataflow.dflow INFO: Task 2229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b5aa98adb4d2d410c360f320e583f4f with result from task 2229 parsl.dataflow.dflow DEBUG: Task 2229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/dbbded6c-6b7f-4744-82dd-22098b8cf86b_isrForCrosstalkSources_1125898_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/dbbded6c-6b7f-4744-82dd-22098b8cf86b_isrForCrosstalkSources_1125898_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:39 proc_lsst.multi:146 [INFO] found job 16896704 in provider checkpoint proc_lsst.multi INFO: found job 16896704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2083 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 049d3ee7dc7ea8b30a0dfb3f60d7b7c1 with result from task 2083 parsl.dataflow.dflow DEBUG: Task 2083 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/aa2fbdad-8f0a-453b-b67b-e3da8b20e751_isrForCrosstalkSources_1125932_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/aa2fbdad-8f0a-453b-b67b-e3da8b20e751_isrForCrosstalkSources_1125932_16.stderr parsl.dataflow.dflow INFO: Task 2157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85f1f067e5dba322190f8bf2cb65dcc4 with result from task 2157 parsl.dataflow.dflow DEBUG: Task 2157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f5f5c472-1872-49fb-aa90-53d0040c21d2_isrForCrosstalkSources_1125910_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f5f5c472-1872-49fb-aa90-53d0040c21d2_isrForCrosstalkSources_1125910_24.stderr parsl.dataflow.dflow INFO: Task 2194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16d490efec073ccccfdf78d578d2879c with result from task 2194 parsl.dataflow.dflow DEBUG: Task 2194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/9510ceaf-98e4-4229-aae0-756886a1a0de_isrForCrosstalkSources_1125940_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/9510ceaf-98e4-4229-aae0-756886a1a0de_isrForCrosstalkSources_1125940_28.stderr parsl.dataflow.dflow INFO: Task 2240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b898f0a4e968506f4254ec7e6764fa1d with result from task 2240 parsl.dataflow.dflow DEBUG: Task 2240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/4ec1f2e6-ca22-4615-a82c-f2d52343e221_isrForCrosstalkSources_1125926_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/4ec1f2e6-ca22-4615-a82c-f2d52343e221_isrForCrosstalkSources_1125926_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:39 proc_lsst.multi:146 [INFO] found job 16896705 in provider checkpoint proc_lsst.multi INFO: found job 16896705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66617b686903f11c97c5ef4ab2285380 with result from task 2160 parsl.dataflow.dflow DEBUG: Task 2160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/9a3173d5-5b06-45c2-9f00-35ad8de7a3ba_isrForCrosstalkSources_1125943_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/9a3173d5-5b06-45c2-9f00-35ad8de7a3ba_isrForCrosstalkSources_1125943_33.stderr parsl.dataflow.dflow INFO: Task 2183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5d1cc3700edff885974f67d7f262929 with result from task 2183 parsl.dataflow.dflow DEBUG: Task 2183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/28c8c9ce-113a-4d41-a2ea-7a5eed38e828_isrForCrosstalkSources_1125936_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/28c8c9ce-113a-4d41-a2ea-7a5eed38e828_isrForCrosstalkSources_1125936_54.stderr parsl.dataflow.dflow INFO: Task 2264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e2d3c6b2edc5c019ed06af33eebf3c2 with result from task 2264 parsl.dataflow.dflow DEBUG: Task 2264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/05ac66f5-7ac2-49f5-8cfb-a975ecd3e360_isrForCrosstalkSources_1125941_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/05ac66f5-7ac2-49f5-8cfb-a975ecd3e360_isrForCrosstalkSources_1125941_35.stderr parsl.dataflow.dflow INFO: Task 2177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20fc1aaa270d9372a69860377aa8f1f5 with result from task 2177 parsl.dataflow.dflow DEBUG: Task 2177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/ad19ac51-23dc-4414-942d-71032bd71166_isrForCrosstalkSources_1125898_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/ad19ac51-23dc-4414-942d-71032bd71166_isrForCrosstalkSources_1125898_4.stderr parsl.dataflow.dflow INFO: Task 2239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51e9eea27e39d9377ca477b42326bb87 with result from task 2239 parsl.dataflow.dflow DEBUG: Task 2239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/f18a1aa7-ce89-43a4-bc40-03660a92d57e_isrForCrosstalkSources_1125890_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/f18a1aa7-ce89-43a4-bc40-03660a92d57e_isrForCrosstalkSources_1125890_12.stderr parsl.dataflow.dflow INFO: Task 2253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2cef90c258132decd43f22d3d330979 with result from task 2253 parsl.dataflow.dflow DEBUG: Task 2253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/73523e99-e0e7-4470-a814-aec7cefb0f5b_isrForCrosstalkSources_1125925_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/73523e99-e0e7-4470-a814-aec7cefb0f5b_isrForCrosstalkSources_1125925_6.stderr parsl.dataflow.dflow INFO: Task 2206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 328ab20a9d65a98056dc9648f4856320 with result from task 2206 parsl.dataflow.dflow DEBUG: Task 2206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/cba191d9-2c1f-458d-9116-c17cda5c45f0_isrForCrosstalkSources_1125935_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/cba191d9-2c1f-458d-9116-c17cda5c45f0_isrForCrosstalkSources_1125935_30.stderr parsl.dataflow.dflow INFO: Task 2190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b0672bac320966c62ab323fa4be2528 with result from task 2190 parsl.dataflow.dflow DEBUG: Task 2190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/77252619-df92-4a89-8f6a-d0abe80cdde0_isrForCrosstalkSources_1125925_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/77252619-df92-4a89-8f6a-d0abe80cdde0_isrForCrosstalkSources_1125925_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:40 proc_lsst.multi:146 [INFO] found job 16896706 in provider checkpoint proc_lsst.multi INFO: found job 16896706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22c8ae1ecef5222ebb37d714815f02ee with result from task 2189 parsl.dataflow.dflow DEBUG: Task 2189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c977e1ca-891d-447a-931c-0bde7de6ae42_isrForCrosstalkSources_1125916_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c977e1ca-891d-447a-931c-0bde7de6ae42_isrForCrosstalkSources_1125916_47.stderr parsl.dataflow.dflow INFO: Task 2207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c14d25756ad42c367fee791edd7c1c53 with result from task 2207 parsl.dataflow.dflow DEBUG: Task 2207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/fa62127b-a001-4f82-a40f-7c0b4356163d_isrForCrosstalkSources_1125905_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/fa62127b-a001-4f82-a40f-7c0b4356163d_isrForCrosstalkSources_1125905_11.stderr parsl.dataflow.dflow INFO: Task 2111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c676703f697e3b6333436ff1555d59ee with result from task 2111 parsl.dataflow.dflow DEBUG: Task 2111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/988cf483-a1db-4948-84ec-b23b2bdda722_isrForCrosstalkSources_1125947_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/988cf483-a1db-4948-84ec-b23b2bdda722_isrForCrosstalkSources_1125947_59.stderr parsl.dataflow.dflow INFO: Task 2235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b993033872b45556f6aa6fcd8de2f6c9 with result from task 2235 parsl.dataflow.dflow DEBUG: Task 2235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/68adccd4-ff84-4dbd-a76c-dc81351204c9_isrForCrosstalkSources_1125923_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/68adccd4-ff84-4dbd-a76c-dc81351204c9_isrForCrosstalkSources_1125923_10.stderr parsl.dataflow.dflow INFO: Task 2089 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d866e31d48a102e7d10cb1e393df33a9 with result from task 2089 parsl.dataflow.dflow DEBUG: Task 2089 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/9e15afb7-f658-492b-b6de-8b4d489ca7dc_isrForCrosstalkSources_1125946_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/9e15afb7-f658-492b-b6de-8b4d489ca7dc_isrForCrosstalkSources_1125946_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:40 proc_lsst.multi:146 [INFO] found job 16896707 in provider checkpoint proc_lsst.multi INFO: found job 16896707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fb439c6e1030760a343574360e782f2 with result from task 2185 parsl.dataflow.dflow DEBUG: Task 2185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/c6b86485-5380-4e2c-93c4-5042d52d40ae_isrForCrosstalkSources_1125918_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/c6b86485-5380-4e2c-93c4-5042d52d40ae_isrForCrosstalkSources_1125918_43.stderr parsl.dataflow.dflow INFO: Task 2247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 276dcf7a3758ccffb999c644b81e79cf with result from task 2247 parsl.dataflow.dflow DEBUG: Task 2247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/e929ff0f-49b3-4481-bd32-78813f590ea8_isrForCrosstalkSources_1125894_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/e929ff0f-49b3-4481-bd32-78813f590ea8_isrForCrosstalkSources_1125894_25.stderr parsl.dataflow.dflow INFO: Task 2102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4448498643a4cf067c851c6b43c66bd1 with result from task 2102 parsl.dataflow.dflow DEBUG: Task 2102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/09f62f20-3fc9-45a9-a2dc-85b74400442e_isrForCrosstalkSources_1125906_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/09f62f20-3fc9-45a9-a2dc-85b74400442e_isrForCrosstalkSources_1125906_8.stderr parsl.dataflow.dflow INFO: Task 2106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bb9a09572f0d084245a700b4e92ac9c with result from task 2106 parsl.dataflow.dflow DEBUG: Task 2106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ed1724cf-d5b5-4f78-a656-f31fe3c4d57d_isrForCrosstalkSources_1125947_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ed1724cf-d5b5-4f78-a656-f31fe3c4d57d_isrForCrosstalkSources_1125947_40.stderr parsl.dataflow.dflow INFO: Task 2219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b640a27a92a06de8bdf2aec28f521ffd with result from task 2219 parsl.dataflow.dflow DEBUG: Task 2219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/c8b29878-a519-4bc2-8f4d-646eb4341469_isrForCrosstalkSources_1125919_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/c8b29878-a519-4bc2-8f4d-646eb4341469_isrForCrosstalkSources_1125919_54.stderr parsl.dataflow.dflow INFO: Task 2131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cf9204a99f870424bd72c4d7c2aa0db with result from task 2131 parsl.dataflow.dflow DEBUG: Task 2131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/aad77c46-c611-4c17-9f14-f6bcf561b0d6_isrForCrosstalkSources_1125904_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/aad77c46-c611-4c17-9f14-f6bcf561b0d6_isrForCrosstalkSources_1125904_21.stderr parsl.dataflow.dflow INFO: Task 2200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39253c3af398e7e5d06d6f4e865b51c5 with result from task 2200 parsl.dataflow.dflow DEBUG: Task 2200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/6837769e-7b51-42ec-92af-2088c0bf3af5_isrForCrosstalkSources_1125945_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/6837769e-7b51-42ec-92af-2088c0bf3af5_isrForCrosstalkSources_1125945_19.stderr parsl.dataflow.dflow INFO: Task 2112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b36a7d35dfafc30f952389062d479cc9 with result from task 2112 parsl.dataflow.dflow DEBUG: Task 2112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/dea42058-860c-47fe-9c0c-fe30c08e1164_isrForCrosstalkSources_1125926_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/dea42058-860c-47fe-9c0c-fe30c08e1164_isrForCrosstalkSources_1125926_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:40 proc_lsst.multi:146 [INFO] found job 16896708 in provider checkpoint proc_lsst.multi INFO: found job 16896708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85bde8a9d54ee5cebaa04fbb40f01d5b with result from task 2201 parsl.dataflow.dflow DEBUG: Task 2201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/8f4d6fcb-287a-4026-aa28-74642537911b_isrForCrosstalkSources_1125937_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/8f4d6fcb-287a-4026-aa28-74642537911b_isrForCrosstalkSources_1125937_17.stderr parsl.dataflow.dflow INFO: Task 2232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e596e3581d199518151f47930af4144 with result from task 2232 parsl.dataflow.dflow DEBUG: Task 2232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/f97b48ac-29d0-4090-9717-d2056eb90e06_isrForCrosstalkSources_1125905_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/f97b48ac-29d0-4090-9717-d2056eb90e06_isrForCrosstalkSources_1125905_30.stderr parsl.dataflow.dflow INFO: Task 2107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40b0154f64cef0ce535c45c0831cdad6 with result from task 2107 parsl.dataflow.dflow DEBUG: Task 2107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/20b9da95-089f-47fe-b64d-51be3f305188_isrForCrosstalkSources_1125942_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/20b9da95-089f-47fe-b64d-51be3f305188_isrForCrosstalkSources_1125942_40.stderr parsl.dataflow.dflow INFO: Task 2214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01a23953abea283eeb21e273c9a2876f with result from task 2214 parsl.dataflow.dflow DEBUG: Task 2214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/cb377207-5885-4fe0-9f08-6fff2f233371_isrForCrosstalkSources_1125917_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/cb377207-5885-4fe0-9f08-6fff2f233371_isrForCrosstalkSources_1125917_47.stderr parsl.dataflow.dflow INFO: Task 2031 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8be64bbd08d3a48b55ea4385b54f5b74 with result from task 2031 parsl.dataflow.dflow DEBUG: Task 2031 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/3d101bb2-d9a1-4bab-a811-41ead43fd38c_isrForCrosstalkSources_1125911_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/3d101bb2-d9a1-4bab-a811-41ead43fd38c_isrForCrosstalkSources_1125911_44.stderr parsl.dataflow.dflow INFO: Task 2244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c65fd24c1b654bb0934dd165a0bfa1a0 with result from task 2244 parsl.dataflow.dflow DEBUG: Task 2244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/02c42c5e-cde6-48ba-b8bb-2c7ac7a44d15_isrForCrosstalkSources_1125902_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/02c42c5e-cde6-48ba-b8bb-2c7ac7a44d15_isrForCrosstalkSources_1125902_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:40 proc_lsst.multi:146 [INFO] found job 16896709 in provider checkpoint proc_lsst.multi INFO: found job 16896709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29707f5bba3009859a1549667970db04 with result from task 2245 parsl.dataflow.dflow DEBUG: Task 2245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/46da1284-ce8e-4605-bf87-ebdcd22fc2ac_isrForCrosstalkSources_1125923_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/46da1284-ce8e-4605-bf87-ebdcd22fc2ac_isrForCrosstalkSources_1125923_15.stderr parsl.dataflow.dflow INFO: Task 2263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2377640cf5dc91ecdd86fd5ea9fb796 with result from task 2263 parsl.dataflow.dflow DEBUG: Task 2263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/a93c81cb-185b-43f9-9c92-6a52f0139315_isrForCrosstalkSources_1125889_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/a93c81cb-185b-43f9-9c92-6a52f0139315_isrForCrosstalkSources_1125889_21.stderr parsl.dataflow.dflow INFO: Task 2174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c776044168adbfa1b41dc44ddd3b8381 with result from task 2174 parsl.dataflow.dflow DEBUG: Task 2174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/6c2e1c86-bb1d-484f-9c98-7e58355bef93_isrForCrosstalkSources_1125928_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/6c2e1c86-bb1d-484f-9c98-7e58355bef93_isrForCrosstalkSources_1125928_24.stderr parsl.dataflow.dflow INFO: Task 2092 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d301df70951f8be56abd9253b22e9a5a with result from task 2092 parsl.dataflow.dflow DEBUG: Task 2092 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/888de858-75b8-4039-ab20-d9ce03792f79_isrForCrosstalkSources_1125890_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/888de858-75b8-4039-ab20-d9ce03792f79_isrForCrosstalkSources_1125890_11.stderr parsl.dataflow.dflow INFO: Task 2122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd608375af68913f8bdb1c8100ec9146 with result from task 2122 parsl.dataflow.dflow DEBUG: Task 2122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/536c6d06-e71c-4a0b-aaf0-bb5071c7e13d_isrForCrosstalkSources_1125904_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/536c6d06-e71c-4a0b-aaf0-bb5071c7e13d_isrForCrosstalkSources_1125904_38.stderr parsl.dataflow.dflow INFO: Task 2237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8bf4150a64b461a1d86d0c476c74b44 with result from task 2237 parsl.dataflow.dflow DEBUG: Task 2237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/4e2ed624-66b8-444c-adaa-e20ed8c35061_isrForCrosstalkSources_1125909_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/4e2ed624-66b8-444c-adaa-e20ed8c35061_isrForCrosstalkSources_1125909_15.stderr parsl.dataflow.dflow INFO: Task 2258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 295e01e91d547f5f4a75d0975c47fbfa with result from task 2258 parsl.dataflow.dflow DEBUG: Task 2258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/c8bacd72-6de5-4afa-93f6-1838d804fb00_isrForCrosstalkSources_1125929_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/c8bacd72-6de5-4afa-93f6-1838d804fb00_isrForCrosstalkSources_1125929_8.stderr parsl.dataflow.dflow INFO: Task 2139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c1e73fc652074d24fcc0b710361421b with result from task 2139 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/84a444d8-4663-454f-9b66-0b85f2a38731_isrForCrosstalkSources_1125928_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/84a444d8-4663-454f-9b66-0b85f2a38731_isrForCrosstalkSources_1125928_41.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:40 proc_lsst.multi:146 [INFO] found job 16896710 in provider checkpoint proc_lsst.multi INFO: found job 16896710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c751a58ba621ab794b28660b681e961b with result from task 2019 parsl.dataflow.dflow DEBUG: Task 2019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/b8f84b9b-6db4-428b-b260-1173b7a41c5b_isrForCrosstalkSources_1125939_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/b8f84b9b-6db4-428b-b260-1173b7a41c5b_isrForCrosstalkSources_1125939_60.stderr parsl.dataflow.dflow INFO: Task 2187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2768c137c2ddae25ce7a46976a0d153 with result from task 2187 parsl.dataflow.dflow DEBUG: Task 2187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/0cad6096-be2a-4f4f-af27-2e97573cf7c7_isrForCrosstalkSources_1125900_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/0cad6096-be2a-4f4f-af27-2e97573cf7c7_isrForCrosstalkSources_1125900_6.stderr parsl.dataflow.dflow INFO: Task 2203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af106c970167349646b87ec0563d139d with result from task 2203 parsl.dataflow.dflow DEBUG: Task 2203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/8f69aa2f-00db-45b1-aea9-bfa2b2e924d3_isrForCrosstalkSources_1125890_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/8f69aa2f-00db-45b1-aea9-bfa2b2e924d3_isrForCrosstalkSources_1125890_26.stderr parsl.dataflow.dflow INFO: Task 2109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3303a0d1c77ddf319a24f6b1033caaf with result from task 2109 parsl.dataflow.dflow DEBUG: Task 2109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/119c123a-b786-4568-b22b-8a034403c63f_isrForCrosstalkSources_1125903_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/119c123a-b786-4568-b22b-8a034403c63f_isrForCrosstalkSources_1125903_17.stderr parsl.dataflow.dflow INFO: Task 2257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f5e348d668616d439c965416f73c1e3 with result from task 2257 parsl.dataflow.dflow DEBUG: Task 2257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/4bff4953-7c48-4312-9892-d295c27b9a5a_isrForCrosstalkSources_1125895_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/4bff4953-7c48-4312-9892-d295c27b9a5a_isrForCrosstalkSources_1125895_21.stderr parsl.dataflow.dflow INFO: Task 2153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3abc721a8c7aed5e29d411af1e372a2 with result from task 2153 parsl.dataflow.dflow DEBUG: Task 2153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/99fba34e-acc7-4ef1-9a50-77da97b1da33_isrForCrosstalkSources_1125931_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/99fba34e-acc7-4ef1-9a50-77da97b1da33_isrForCrosstalkSources_1125931_1.stderr parsl.dataflow.dflow INFO: Task 2094 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b09d308f2e3832809ad1387a3ae54fd with result from task 2094 parsl.dataflow.dflow DEBUG: Task 2094 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/ee448847-32a4-4559-9529-9a7ab02de663_isrForCrosstalkSources_1125928_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/ee448847-32a4-4559-9529-9a7ab02de663_isrForCrosstalkSources_1125928_61.stderr parsl.dataflow.dflow INFO: Task 2249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df850f894a5de4d2ce01410db0765195 with result from task 2249 parsl.dataflow.dflow DEBUG: Task 2249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/bd1b1e3d-41a0-4574-8237-fcadb3bd0721_isrForCrosstalkSources_1125927_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/bd1b1e3d-41a0-4574-8237-fcadb3bd0721_isrForCrosstalkSources_1125927_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:41 proc_lsst.multi:146 [INFO] found job 16896711 in provider checkpoint proc_lsst.multi INFO: found job 16896711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34fcd22cd7cd6e9899f9af19701296ab with result from task 2114 parsl.dataflow.dflow DEBUG: Task 2114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/35d0c911-c224-42de-8df2-c5bb7a8efa94_isrForCrosstalkSources_1125907_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/35d0c911-c224-42de-8df2-c5bb7a8efa94_isrForCrosstalkSources_1125907_49.stderr parsl.dataflow.dflow INFO: Task 2268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dbfc1a1aa6c15927254b20d85ef2727 with result from task 2268 parsl.dataflow.dflow DEBUG: Task 2268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/dbf09398-dbb1-4903-9645-40abba962ac2_isrForCrosstalkSources_1125900_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/dbf09398-dbb1-4903-9645-40abba962ac2_isrForCrosstalkSources_1125900_7.stderr parsl.dataflow.dflow INFO: Task 2086 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bfee53454eaa798a67335ad09a09136 with result from task 2086 parsl.dataflow.dflow DEBUG: Task 2086 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/68fd3829-9a74-4dbb-b581-98d9eef7236a_isrForCrosstalkSources_1125896_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/68fd3829-9a74-4dbb-b581-98d9eef7236a_isrForCrosstalkSources_1125896_31.stderr parsl.dataflow.dflow INFO: Task 2213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33f260e4165130d0a0af4f696438461c with result from task 2213 parsl.dataflow.dflow DEBUG: Task 2213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/3e578b30-e372-4e90-b4f6-fc30b7149e13_isrForCrosstalkSources_1125922_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/3e578b30-e372-4e90-b4f6-fc30b7149e13_isrForCrosstalkSources_1125922_8.stderr parsl.dataflow.dflow INFO: Task 2178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24316e3e8e8e73af32b39c4b137d2c37 with result from task 2178 parsl.dataflow.dflow DEBUG: Task 2178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/d528b682-c3a5-4492-86f7-701eadcf4d6d_isrForCrosstalkSources_1125903_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/d528b682-c3a5-4492-86f7-701eadcf4d6d_isrForCrosstalkSources_1125903_44.stderr parsl.dataflow.dflow INFO: Task 2220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c872779d27ac8f822ef7e5952579b5e0 with result from task 2220 parsl.dataflow.dflow DEBUG: Task 2220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/2782a268-894f-453d-93d5-f7148dfc88bd_isrForCrosstalkSources_1125890_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/2782a268-894f-453d-93d5-f7148dfc88bd_isrForCrosstalkSources_1125890_56.stderr parsl.dataflow.dflow INFO: Task 2260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cea27635e9c325d7603afb6d7daaf485 with result from task 2260 parsl.dataflow.dflow DEBUG: Task 2260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/0c564588-c751-4677-8ef0-dcedbeee7f4a_isrForCrosstalkSources_1125922_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/0c564588-c751-4677-8ef0-dcedbeee7f4a_isrForCrosstalkSources_1125922_34.stderr parsl.dataflow.dflow INFO: Task 2085 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1af12ae482ced5e2fae4c48998f925d with result from task 2085 parsl.dataflow.dflow DEBUG: Task 2085 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/9d047b1b-e17a-4c44-b1d7-52e4d6874f0f_isrForCrosstalkSources_1125903_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/9d047b1b-e17a-4c44-b1d7-52e4d6874f0f_isrForCrosstalkSources_1125903_49.stderr parsl.dataflow.dflow INFO: Task 2125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e01cc8f2c3c49f207679ce341718c686 with result from task 2125 parsl.dataflow.dflow DEBUG: Task 2125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/f74b16f7-7226-4668-8a85-e45dc717de34_isrForCrosstalkSources_1125903_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/f74b16f7-7226-4668-8a85-e45dc717de34_isrForCrosstalkSources_1125903_52.stderr parsl.dataflow.dflow INFO: Task 2250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dbfd04bb1665229bc12183fa11ce878 with result from task 2250 parsl.dataflow.dflow DEBUG: Task 2250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/698fd1f7-7bfc-4154-af3d-6eea431ae838_isrForCrosstalkSources_1125910_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/698fd1f7-7bfc-4154-af3d-6eea431ae838_isrForCrosstalkSources_1125910_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abff6a2086e8b3a2fde6b3b42fced00d with result from task 2163 parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:41 proc_lsst.multi:146 [INFO] found job 16896712 in provider checkpoint proc_lsst.multi INFO: found job 16896712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/2e049568-a9c5-4b12-8406-0f6a302bb964_isrForCrosstalkSources_1125929_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/2e049568-a9c5-4b12-8406-0f6a302bb964_isrForCrosstalkSources_1125929_60.stderr parsl.dataflow.dflow INFO: Task 2259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b92fb6741ee855a4032be7cfa44a721 with result from task 2259 parsl.dataflow.dflow DEBUG: Task 2259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/32ffbda3-cd92-4317-912b-1522ab435a3f_isrForCrosstalkSources_1125888_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/32ffbda3-cd92-4317-912b-1522ab435a3f_isrForCrosstalkSources_1125888_15.stderr parsl.dataflow.dflow INFO: Task 2172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6d579056df14c4709131dbc60fa9deb with result from task 2172 parsl.dataflow.dflow DEBUG: Task 2172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c2dad2df-778b-4337-b4ec-954a7030f830_isrForCrosstalkSources_1125917_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c2dad2df-778b-4337-b4ec-954a7030f830_isrForCrosstalkSources_1125917_30.stderr parsl.dataflow.dflow INFO: Task 2267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96ba0ee97477ce44d561b67ef1d672b1 with result from task 2267 parsl.dataflow.dflow DEBUG: Task 2267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/696a8951-2349-4f6f-a318-86a78ac2f77e_isrForCrosstalkSources_1125904_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/696a8951-2349-4f6f-a318-86a78ac2f77e_isrForCrosstalkSources_1125904_37.stderr parsl.dataflow.dflow INFO: Task 2165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a9b9175e0b1d9cf68dd05cc4f7b4056 with result from task 2165 parsl.dataflow.dflow DEBUG: Task 2165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4feee85d-5800-4c4a-b649-31fc10198f9d_isrForCrosstalkSources_1125908_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4feee85d-5800-4c4a-b649-31fc10198f9d_isrForCrosstalkSources_1125908_55.stderr parsl.dataflow.dflow INFO: Task 2169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6b49850f5e5bd6fb1c9cd291dbfb5a5 with result from task 2169 parsl.dataflow.dflow DEBUG: Task 2169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/31e68a55-2db6-4cb1-bbb0-4210ce048c7f_isrForCrosstalkSources_1125917_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/31e68a55-2db6-4cb1-bbb0-4210ce048c7f_isrForCrosstalkSources_1125917_62.stderr parsl.dataflow.dflow INFO: Task 2049 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e145cbc0ee8d164e625def4bbb3ae26 with result from task 2049 parsl.dataflow.dflow DEBUG: Task 2049 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/bf7a9259-4c89-48fc-9e16-bd9f2769890d_isrForCrosstalkSources_1125946_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/bf7a9259-4c89-48fc-9e16-bd9f2769890d_isrForCrosstalkSources_1125946_47.stderr parsl.dataflow.dflow INFO: Task 2167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab51807121218a4ebd9c0b75487f6b7a with result from task 2167 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:41 proc_lsst.multi:146 [INFO] found job 16896713 in provider checkpoint proc_lsst.multi INFO: found job 16896713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/cc3fe9bb-98ac-4af5-850f-ffa95d38ba2e_isrForCrosstalkSources_1125891_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/cc3fe9bb-98ac-4af5-850f-ffa95d38ba2e_isrForCrosstalkSources_1125891_60.stderr parsl.dataflow.dflow INFO: Task 2242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 919a80cc1eb14b630629ca5f171977ea with result from task 2242 parsl.dataflow.dflow DEBUG: Task 2242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/3609bc73-cc3c-4fb6-947e-3f2e7c15f520_isrForCrosstalkSources_1125925_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/3609bc73-cc3c-4fb6-947e-3f2e7c15f520_isrForCrosstalkSources_1125925_58.stderr parsl.dataflow.dflow INFO: Task 2155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36a7144b94b18255abded858c71721ef with result from task 2155 parsl.dataflow.dflow DEBUG: Task 2155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/7dba3dd4-6f6e-4e7f-aa5c-3da54cb29dcc_isrForCrosstalkSources_1125932_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/7dba3dd4-6f6e-4e7f-aa5c-3da54cb29dcc_isrForCrosstalkSources_1125932_43.stderr parsl.dataflow.dflow INFO: Task 2095 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d392aa2391d5a9f7f14e78ec2c0a586c with result from task 2095 parsl.dataflow.dflow DEBUG: Task 2095 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/83fa7603-3741-4e2c-ac84-490193a63fe9_isrForCrosstalkSources_1125947_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/83fa7603-3741-4e2c-ac84-490193a63fe9_isrForCrosstalkSources_1125947_62.stderr parsl.dataflow.dflow INFO: Task 2314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c642101ebb16d17a4aa33fb72490477 with result from task 2314 parsl.dataflow.dflow DEBUG: Task 2314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/26eae107-3227-4056-9775-62ef50a710bb_isrForCrosstalkSources_1125896_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/26eae107-3227-4056-9775-62ef50a710bb_isrForCrosstalkSources_1125896_26.stderr parsl.dataflow.dflow INFO: Task 2279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1a76341370a77e849f19621221c5937 with result from task 2279 parsl.dataflow.dflow DEBUG: Task 2279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/5f8cf3a8-4b5b-47f9-9f67-c806e0b5b34f_isrForCrosstalkSources_1125893_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/5f8cf3a8-4b5b-47f9-9f67-c806e0b5b34f_isrForCrosstalkSources_1125893_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:41 proc_lsst.multi:146 [INFO] found job 16896714 in provider checkpoint proc_lsst.multi INFO: found job 16896714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe478c6ea0343d7164154a9e4f268d46 with result from task 2273 parsl.dataflow.dflow DEBUG: Task 2273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/126e08cd-97cc-4589-9e30-cd9cb9f806d2_isrForCrosstalkSources_1125893_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/126e08cd-97cc-4589-9e30-cd9cb9f806d2_isrForCrosstalkSources_1125893_39.stderr parsl.dataflow.dflow INFO: Task 2151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d9449e3dc44142e384ae722456c3d51 with result from task 2151 parsl.dataflow.dflow DEBUG: Task 2151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/3caf0e6c-5069-444c-8fb6-3d46aa228f71_isrForCrosstalkSources_1125915_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/3caf0e6c-5069-444c-8fb6-3d46aa228f71_isrForCrosstalkSources_1125915_53.stderr parsl.dataflow.dflow INFO: Task 2286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f46aa2c7cbe40220dbc7da741662d8f3 with result from task 2286 parsl.dataflow.dflow DEBUG: Task 2286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/6de77a45-14af-4df0-861e-dbc58d28a3a0_isrForCrosstalkSources_1125930_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/6de77a45-14af-4df0-861e-dbc58d28a3a0_isrForCrosstalkSources_1125930_31.stderr parsl.dataflow.dflow INFO: Task 2281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c693f483a7b34486fe20e672627b36b7 with result from task 2281 parsl.dataflow.dflow DEBUG: Task 2281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/3581eb79-7415-4797-9b40-0deb2466b66d_isrForCrosstalkSources_1125918_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/3581eb79-7415-4797-9b40-0deb2466b66d_isrForCrosstalkSources_1125918_11.stderr parsl.dataflow.dflow INFO: Task 2298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8897b94472d92c5c9b392bf20430a849 with result from task 2298 parsl.dataflow.dflow DEBUG: Task 2298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/bc5e32d3-2bbf-484c-be60-c4fc25cf70be_isrForCrosstalkSources_1125891_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/bc5e32d3-2bbf-484c-be60-c4fc25cf70be_isrForCrosstalkSources_1125891_21.stderr parsl.dataflow.dflow INFO: Task 2116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 068d5ca7bebbde2c8441ea9686865f87 with result from task 2116 parsl.dataflow.dflow DEBUG: Task 2116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/349fc28a-a905-4b6a-b665-3a8517869639_isrForCrosstalkSources_1125938_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/349fc28a-a905-4b6a-b665-3a8517869639_isrForCrosstalkSources_1125938_42.stderr parsl.dataflow.dflow INFO: Task 2265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3770f217811ad91533e95b52f30b461 with result from task 2265 parsl.dataflow.dflow DEBUG: Task 2265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e5901df3-5814-4214-9d94-c14e00fdc230_isrForCrosstalkSources_1125929_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e5901df3-5814-4214-9d94-c14e00fdc230_isrForCrosstalkSources_1125929_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 312, active_tasks = 1482 parsl.dataflow.strategy DEBUG: Executor multi has 1482 active tasks, 41/271 running/pending blocks, and 291 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2a: active_blocks 312 >= max_blocks 312 so not scaling out parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22377792571472 2024-03-07 15:13:42 proc_lsst.multi:146 [INFO] found job 53087 in provider local proc_lsst.multi INFO: found job 53087 in provider local 2024-03-07 15:13:42 proc_lsst.multi:146 [INFO] found job 16896385 in provider astro proc_lsst.multi INFO: found job 16896385 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3c7bd77b3ef5b6305bbf6e0c9af7cff with result from task 2100 parsl.dataflow.dflow DEBUG: Task 2100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/064cff44-050d-4f0e-9dd8-2b46b56f58fe_isrForCrosstalkSources_1125912_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/064cff44-050d-4f0e-9dd8-2b46b56f58fe_isrForCrosstalkSources_1125912_50.stderr parsl.dataflow.dflow INFO: Task 2124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06dfa003f904d9a7f98acac17477e485 with result from task 2124 parsl.dataflow.dflow DEBUG: Task 2124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/9a1ac523-c23c-4bba-98d3-be41cbbbd3b8_isrForCrosstalkSources_1125914_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/9a1ac523-c23c-4bba-98d3-be41cbbbd3b8_isrForCrosstalkSources_1125914_59.stderr parsl.dataflow.dflow INFO: Task 2299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24b4c8f4093a159b975314e90f17c0da with result from task 2299 parsl.dataflow.dflow DEBUG: Task 2299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/47a7def7-30e7-4014-839b-97b04201129f_isrForCrosstalkSources_1125914_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/47a7def7-30e7-4014-839b-97b04201129f_isrForCrosstalkSources_1125914_40.stderr parsl.dataflow.dflow INFO: Task 2134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cca9ca02bccb3bbf35de6ebe8a60f91 with result from task 2134 parsl.dataflow.dflow DEBUG: Task 2134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/8ae05ca8-85d4-4c42-945a-6e7c8f78ef8c_isrForCrosstalkSources_1125922_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/8ae05ca8-85d4-4c42-945a-6e7c8f78ef8c_isrForCrosstalkSources_1125922_57.stderr parsl.dataflow.dflow INFO: Task 2113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b5f465879b82cad4b35ad6e6b98cb05 with result from task 2113 parsl.dataflow.dflow DEBUG: Task 2113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/4130db13-faab-4b8e-af99-1919fcddf50e_isrForCrosstalkSources_1125934_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/4130db13-faab-4b8e-af99-1919fcddf50e_isrForCrosstalkSources_1125934_60.stderr parsl.dataflow.dflow INFO: Task 2309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1266094524b2e18e0b7528f2dc2a16a3 with result from task 2309 parsl.dataflow.dflow DEBUG: Task 2309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/7cc750a4-830d-40a2-9478-a1cc8f7a4e37_isrForCrosstalkSources_1125910_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/7cc750a4-830d-40a2-9478-a1cc8f7a4e37_isrForCrosstalkSources_1125910_40.stderr parsl.dataflow.dflow INFO: Task 2295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 112ce67af24992d1c6e9bb18f8440364 with result from task 2295 parsl.dataflow.dflow DEBUG: Task 2295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/1840d645-6d91-4d7b-b9fc-eb8867866769_isrForCrosstalkSources_1125917_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/1840d645-6d91-4d7b-b9fc-eb8867866769_isrForCrosstalkSources_1125917_51.stderr parsl.dataflow.dflow INFO: Task 2234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c13a554b7f03edbe71bbcf7f37ca3d5 with result from task 2234 parsl.dataflow.dflow DEBUG: Task 2234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/b2b0c46d-2837-4a8b-a0e7-d14c70f9fe45_isrForCrosstalkSources_1125945_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/b2b0c46d-2837-4a8b-a0e7-d14c70f9fe45_isrForCrosstalkSources_1125945_56.stderr parsl.dataflow.dflow INFO: Task 2210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4f2c13452f047471e81a23e44ff7be3 with result from task 2210 parsl.dataflow.dflow DEBUG: Task 2210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/0d050859-d80a-42d9-98bf-0ec0aecb9bcd_isrForCrosstalkSources_1125925_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/0d050859-d80a-42d9-98bf-0ec0aecb9bcd_isrForCrosstalkSources_1125925_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:42 proc_lsst.multi:146 [INFO] found job 16896386 in provider astro proc_lsst.multi INFO: found job 16896386 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8137e4015c5b0dcd3985e3a31ca136f0 with result from task 2176 parsl.dataflow.dflow DEBUG: Task 2176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/29cb678b-5127-4e34-87f5-3a797992e083_isrForCrosstalkSources_1125899_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/29cb678b-5127-4e34-87f5-3a797992e083_isrForCrosstalkSources_1125899_58.stderr parsl.dataflow.dflow INFO: Task 2292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdc317df28589c1bf41adb297043f63a with result from task 2292 parsl.dataflow.dflow DEBUG: Task 2292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/f92f1455-22b5-40c8-a812-3b179cfa8cc4_isrForCrosstalkSources_1125929_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/f92f1455-22b5-40c8-a812-3b179cfa8cc4_isrForCrosstalkSources_1125929_26.stderr parsl.dataflow.dflow INFO: Task 2290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 211bdda86ab39e104576a681491fc830 with result from task 2290 parsl.dataflow.dflow DEBUG: Task 2290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/b16391ed-d4b2-4c79-9ab6-6d5d241c8f49_isrForCrosstalkSources_1125930_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/b16391ed-d4b2-4c79-9ab6-6d5d241c8f49_isrForCrosstalkSources_1125930_26.stderr parsl.dataflow.dflow INFO: Task 2287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de2d98d971466bc7e212c37d4e6705f0 with result from task 2287 parsl.dataflow.dflow DEBUG: Task 2287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f9914589-add5-4e71-ad98-ef3ac5cd9f75_isrForCrosstalkSources_1125945_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f9914589-add5-4e71-ad98-ef3ac5cd9f75_isrForCrosstalkSources_1125945_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:42 proc_lsst.multi:146 [INFO] found job 16896387 in provider astro proc_lsst.multi INFO: found job 16896387 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:42 proc_lsst.multi:146 [INFO] found job 16896388 in provider astro proc_lsst.multi INFO: found job 16896388 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:43 proc_lsst.multi:146 [INFO] found job 16896389 in provider astro proc_lsst.multi INFO: found job 16896389 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbb400e309b1ef8ab5f9feedad1bf18f with result from task 2310 parsl.dataflow.dflow DEBUG: Task 2310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/98f9cf9e-2db5-4965-8abe-6977f3ffa81a_isrForCrosstalkSources_1125896_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/98f9cf9e-2db5-4965-8abe-6977f3ffa81a_isrForCrosstalkSources_1125896_29.stderr parsl.dataflow.dflow INFO: Task 2308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d552b6e63d7f9e5cac214382ce2efa74 with result from task 2308 parsl.dataflow.dflow DEBUG: Task 2308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/550d89f0-486e-4b3c-9f73-427a6d5bb970_isrForCrosstalkSources_1125944_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/550d89f0-486e-4b3c-9f73-427a6d5bb970_isrForCrosstalkSources_1125944_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:43 proc_lsst.multi:146 [INFO] found job 16896390 in provider astro proc_lsst.multi INFO: found job 16896390 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05f3b8f8330dcb0a1dfcbeb88fd22b5d with result from task 2280 parsl.dataflow.dflow DEBUG: Task 2280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b2035a2a-10b5-4e8a-b363-2b1d18443eb4_isrForCrosstalkSources_1125938_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b2035a2a-10b5-4e8a-b363-2b1d18443eb4_isrForCrosstalkSources_1125938_5.stderr parsl.dataflow.dflow INFO: Task 2274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14e752c9db165e7cd142ab9e70c47b52 with result from task 2274 parsl.dataflow.dflow DEBUG: Task 2274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/03ddd70e-0cad-4f6a-933d-702eba20b03a_isrForCrosstalkSources_1125933_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/03ddd70e-0cad-4f6a-933d-702eba20b03a_isrForCrosstalkSources_1125933_51.stderr parsl.dataflow.dflow INFO: Task 2297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 598086ebddc98da5523b29f8ce664361 with result from task 2297 parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:43 proc_lsst.multi:146 [INFO] found job 16896392 in provider astro proc_lsst.multi INFO: found job 16896392 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow DEBUG: Task 2297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/4a0ca294-1a79-4096-80fc-dd9cacacee8d_isrForCrosstalkSources_1125929_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/4a0ca294-1a79-4096-80fc-dd9cacacee8d_isrForCrosstalkSources_1125929_32.stderr parsl.dataflow.dflow INFO: Task 2327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10efe3fe124d7fd44ba5ae43a78bbcff with result from task 2327 parsl.dataflow.dflow DEBUG: Task 2327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/5bf7d879-7051-4c77-8d74-6d12ef6eefc0_isrForCrosstalkSources_1125934_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/5bf7d879-7051-4c77-8d74-6d12ef6eefc0_isrForCrosstalkSources_1125934_12.stderr parsl.dataflow.dflow INFO: Task 2352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f93ff64f9aa71c056b96cece69d6735 with result from task 2352 parsl.dataflow.dflow DEBUG: Task 2352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/4f00b3b4-c73c-4c93-a84a-c00813956f0c_isrForCrosstalkSources_1125897_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/4f00b3b4-c73c-4c93-a84a-c00813956f0c_isrForCrosstalkSources_1125897_40.stderr parsl.dataflow.dflow INFO: Task 2322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d937b1c298add24cf7e55634f3e117f with result from task 2322 parsl.dataflow.dflow DEBUG: Task 2322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/b307b780-7483-4481-8236-1c666f314f4e_isrForCrosstalkSources_1125892_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/b307b780-7483-4481-8236-1c666f314f4e_isrForCrosstalkSources_1125892_28.stderr parsl.dataflow.dflow INFO: Task 2328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f38a5e04752dedb4b7ffdeccbeea03d with result from task 2328 parsl.dataflow.dflow DEBUG: Task 2328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/a9c9b639-e2df-43b9-8514-08b06e62d5a0_isrForCrosstalkSources_1125940_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/a9c9b639-e2df-43b9-8514-08b06e62d5a0_isrForCrosstalkSources_1125940_47.stderr parsl.dataflow.dflow INFO: Task 2296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37887d36ad04d7fa7cec6a1b7b39da66 with result from task 2296 parsl.dataflow.dflow DEBUG: Task 2296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/45d047f1-19a8-4a7b-9d90-e9ff55bb1f5f_isrForCrosstalkSources_1125939_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/45d047f1-19a8-4a7b-9d90-e9ff55bb1f5f_isrForCrosstalkSources_1125939_46.stderr parsl.dataflow.dflow INFO: Task 2349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e2748386210171055ab2f059bdeccd0 with result from task 2349 parsl.dataflow.dflow DEBUG: Task 2349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/314c016e-6af3-497b-8c47-5f0175cd47d1_isrForCrosstalkSources_1125916_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/314c016e-6af3-497b-8c47-5f0175cd47d1_isrForCrosstalkSources_1125916_16.stderr parsl.dataflow.dflow INFO: Task 2326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d99917bde3b8b15e03db047a53baeb1 with result from task 2326 parsl.dataflow.dflow DEBUG: Task 2326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/1d8cc44a-5e3e-4719-b064-e88556a49abd_isrForCrosstalkSources_1125917_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/1d8cc44a-5e3e-4719-b064-e88556a49abd_isrForCrosstalkSources_1125917_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:44 proc_lsst.multi:146 [INFO] found job 16896394 in provider astro proc_lsst.multi INFO: found job 16896394 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d888608031bc6283b335ca9d59c5e8d with result from task 2302 parsl.dataflow.dflow DEBUG: Task 2302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/f541d420-fb63-428a-800e-79138f31bed0_isrForCrosstalkSources_1125916_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/f541d420-fb63-428a-800e-79138f31bed0_isrForCrosstalkSources_1125916_41.stderr parsl.dataflow.dflow INFO: Task 2269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 008711a3379cf30f423ce2dc5d22f2d2 with result from task 2269 parsl.dataflow.dflow DEBUG: Task 2269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/726a7f5a-e500-4536-b33e-0f296ef12943_isrForCrosstalkSources_1125922_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/726a7f5a-e500-4536-b33e-0f296ef12943_isrForCrosstalkSources_1125922_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:44 proc_lsst.multi:146 [INFO] found job 16896395 in provider astro proc_lsst.multi INFO: found job 16896395 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f83ab7395e980fa4b544079b0036a1f8 with result from task 2335 parsl.dataflow.dflow DEBUG: Task 2335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/e2803dfc-c1be-4c6f-a3d7-8e95737a6fe9_isrForCrosstalkSources_1125909_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/e2803dfc-c1be-4c6f-a3d7-8e95737a6fe9_isrForCrosstalkSources_1125909_28.stderr parsl.dataflow.dflow INFO: Task 2291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b013285596e2896bb15ce2c3dec19fa with result from task 2291 parsl.dataflow.dflow DEBUG: Task 2291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/8fe1660f-5135-44d4-99dd-430a71a88c7e_isrForCrosstalkSources_1125935_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/8fe1660f-5135-44d4-99dd-430a71a88c7e_isrForCrosstalkSources_1125935_42.stderr parsl.dataflow.dflow INFO: Task 2293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a21207fef6792dbbf088c5f1d0720909 with result from task 2293 parsl.dataflow.dflow DEBUG: Task 2293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/30c456a6-816a-496b-a700-209e7eae47df_isrForCrosstalkSources_1125907_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/30c456a6-816a-496b-a700-209e7eae47df_isrForCrosstalkSources_1125907_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:44 proc_lsst.multi:146 [INFO] found job 16896396 in provider astro proc_lsst.multi INFO: found job 16896396 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecb7bc4f94edf2700c72bc2daa6a1c5d with result from task 2339 parsl.dataflow.dflow DEBUG: Task 2339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/7748ccb5-5ac8-4f92-bb31-54cc18cef6f1_isrForCrosstalkSources_1125907_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/7748ccb5-5ac8-4f92-bb31-54cc18cef6f1_isrForCrosstalkSources_1125907_6.stderr parsl.dataflow.dflow INFO: Task 2300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 138e962192bab77f381aafeac430b312 with result from task 2300 parsl.dataflow.dflow DEBUG: Task 2300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/ad3d6ece-a558-47c5-a4ee-848e39977502_isrForCrosstalkSources_1125933_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/ad3d6ece-a558-47c5-a4ee-848e39977502_isrForCrosstalkSources_1125933_33.stderr parsl.dataflow.dflow INFO: Task 2340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2482a27c8b6496d0db8bbf43851e35f5 with result from task 2340 parsl.dataflow.dflow DEBUG: Task 2340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/8136dcb2-46b3-4a3a-957b-dc56265901b0_isrForCrosstalkSources_1125934_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/8136dcb2-46b3-4a3a-957b-dc56265901b0_isrForCrosstalkSources_1125934_20.stderr parsl.dataflow.dflow INFO: Task 2320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea475d162da7f4197197bd152ffd862d with result from task 2320 parsl.dataflow.dflow DEBUG: Task 2320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/ee4233d2-933b-4ccb-b001-313e28f26ceb_isrForCrosstalkSources_1125890_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/ee4233d2-933b-4ccb-b001-313e28f26ceb_isrForCrosstalkSources_1125890_19.stderr parsl.dataflow.dflow INFO: Task 2385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9b792d5ce914b81869190b83e44e6b4 with result from task 2385 parsl.dataflow.dflow DEBUG: Task 2385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/077330c3-0182-492c-96e1-232058ddeab0_isrForCrosstalkSources_1125900_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/077330c3-0182-492c-96e1-232058ddeab0_isrForCrosstalkSources_1125900_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:44 proc_lsst.multi:146 [INFO] found job 16896397 in provider astro proc_lsst.multi INFO: found job 16896397 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c12a42ba6ba25a7c3cd9c113a31bba14 with result from task 2306 parsl.dataflow.dflow DEBUG: Task 2306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/d35d2e45-3781-4361-97c8-4776afac9826_isrForCrosstalkSources_1125928_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/d35d2e45-3781-4361-97c8-4776afac9826_isrForCrosstalkSources_1125928_48.stderr parsl.dataflow.dflow INFO: Task 2358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3cbb908a62ef5ca6e17356b86c976cf with result from task 2358 parsl.dataflow.dflow DEBUG: Task 2358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/2cd9d7de-7b0e-4326-9b50-85be44407a8d_isrForCrosstalkSources_1125942_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/2cd9d7de-7b0e-4326-9b50-85be44407a8d_isrForCrosstalkSources_1125942_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:45 proc_lsst.multi:146 [INFO] found job 16896398 in provider astro proc_lsst.multi INFO: found job 16896398 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31e1abeb8be6040fd77222a3a4201dc6 with result from task 2254 parsl.dataflow.dflow DEBUG: Task 2254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/632fab6f-4401-4cf6-9acf-352501e22b32_isrForCrosstalkSources_1125904_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/632fab6f-4401-4cf6-9acf-352501e22b32_isrForCrosstalkSources_1125904_31.stderr parsl.dataflow.dflow INFO: Task 2337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2d3b530d5f2ec66cb8dbed0422a923f with result from task 2337 parsl.dataflow.dflow DEBUG: Task 2337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/b752884d-d521-472c-9e8d-add75e8e9a0e_isrForCrosstalkSources_1125945_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/b752884d-d521-472c-9e8d-add75e8e9a0e_isrForCrosstalkSources_1125945_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:45 proc_lsst.multi:146 [INFO] found job 16896399 in provider astro proc_lsst.multi INFO: found job 16896399 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22c9810cb37d1f3fb9f6dd47c8e3f581 with result from task 2317 parsl.dataflow.dflow DEBUG: Task 2317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/95df473d-7618-41b3-af61-6c713950c5bf_isrForCrosstalkSources_1125912_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/95df473d-7618-41b3-af61-6c713950c5bf_isrForCrosstalkSources_1125912_32.stderr parsl.dataflow.dflow INFO: Task 2315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8f9567c8f9f26af1f85b14b571c5d2c with result from task 2315 parsl.dataflow.dflow DEBUG: Task 2315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/ca1b7f6a-6ad7-43d6-9536-3f0e52d0f494_isrForCrosstalkSources_1125937_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/ca1b7f6a-6ad7-43d6-9536-3f0e52d0f494_isrForCrosstalkSources_1125937_13.stderr parsl.dataflow.dflow INFO: Task 2221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84dc8540c5fc6401e34e54a684cfd72a with result from task 2221 parsl.dataflow.dflow DEBUG: Task 2221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/c093de66-9ed2-41d3-b396-78986b6ce3eb_isrForCrosstalkSources_1125920_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/c093de66-9ed2-41d3-b396-78986b6ce3eb_isrForCrosstalkSources_1125920_54.stderr parsl.dataflow.dflow INFO: Task 2277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81e895717d6a09165dc04dff2d0b52a1 with result from task 2277 parsl.dataflow.dflow DEBUG: Task 2277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/481c27ed-d7d3-4ac2-ac41-d0547ef61d0b_isrForCrosstalkSources_1125893_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/481c27ed-d7d3-4ac2-ac41-d0547ef61d0b_isrForCrosstalkSources_1125893_20.stderr parsl.dataflow.dflow INFO: Task 2399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7ea46892178613053133d4d966208ac with result from task 2399 parsl.dataflow.dflow DEBUG: Task 2399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/728871dd-ed06-491f-8e30-4afd55744b15_isrForCrosstalkSources_1125920_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/728871dd-ed06-491f-8e30-4afd55744b15_isrForCrosstalkSources_1125920_57.stderr parsl.dataflow.dflow INFO: Task 2342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3de46d899c0bdee0deed13a3637bf163 with result from task 2342 parsl.dataflow.dflow DEBUG: Task 2342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/188cdf08-3d53-49a9-b7a1-37fba0d6465d_isrForCrosstalkSources_1125925_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/188cdf08-3d53-49a9-b7a1-37fba0d6465d_isrForCrosstalkSources_1125925_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:45 proc_lsst.multi:146 [INFO] found job 16896400 in provider astro proc_lsst.multi INFO: found job 16896400 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b33e8f0f2fca07e4df49c43cb0050d9f with result from task 2294 parsl.dataflow.dflow DEBUG: Task 2294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/80daaa2f-1043-4424-8775-4a3856479206_isrForCrosstalkSources_1125930_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/80daaa2f-1043-4424-8775-4a3856479206_isrForCrosstalkSources_1125930_14.stderr parsl.dataflow.dflow INFO: Task 2270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0b0e4b68c4c61501330e6afd1ec558b with result from task 2270 parsl.dataflow.dflow DEBUG: Task 2270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/84849fdb-565c-4cec-9dbc-597b7c7c8b8f_isrForCrosstalkSources_1125925_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/84849fdb-565c-4cec-9dbc-597b7c7c8b8f_isrForCrosstalkSources_1125925_25.stderr parsl.dataflow.dflow INFO: Task 2343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19cac0c4046829c93743a144612cfbf3 with result from task 2343 parsl.dataflow.dflow DEBUG: Task 2343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/21e66ce4-dcc1-4b39-92ed-1112fb50d9dc_isrForCrosstalkSources_1125913_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/21e66ce4-dcc1-4b39-92ed-1112fb50d9dc_isrForCrosstalkSources_1125913_8.stderr parsl.dataflow.dflow INFO: Task 2266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b1b6bc59f1839a0880caa80685a1f94 with result from task 2266 parsl.dataflow.dflow DEBUG: Task 2266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/264aded8-f06c-45ca-bd90-7a64da8a04b4_isrForCrosstalkSources_1125938_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/264aded8-f06c-45ca-bd90-7a64da8a04b4_isrForCrosstalkSources_1125938_24.stderr parsl.dataflow.dflow INFO: Task 2238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 119986c7c1491a421e48b9899114acab with result from task 2238 parsl.dataflow.dflow DEBUG: Task 2238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/0a3c92c7-2b43-4cae-bc08-0fac9cfc4d62_isrForCrosstalkSources_1125932_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/0a3c92c7-2b43-4cae-bc08-0fac9cfc4d62_isrForCrosstalkSources_1125932_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:45 proc_lsst.multi:146 [INFO] found job 16896401 in provider astro proc_lsst.multi INFO: found job 16896401 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:46 proc_lsst.multi:146 [INFO] found job 16896402 in provider astro proc_lsst.multi INFO: found job 16896402 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b854a8f1cc7ca1b9eaf22914025b010 with result from task 2272 parsl.dataflow.dflow DEBUG: Task 2272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/22b9e9cd-36cb-4d5e-8ae8-7717dd68481f_isrForCrosstalkSources_1125900_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/22b9e9cd-36cb-4d5e-8ae8-7717dd68481f_isrForCrosstalkSources_1125900_12.stderr parsl.dataflow.dflow INFO: Task 2325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc8a1177a7c40cf9154aa937d97751af with result from task 2325 parsl.dataflow.dflow DEBUG: Task 2325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/723dd616-72a4-434a-bcf3-1aed555bf91a_isrForCrosstalkSources_1125914_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/723dd616-72a4-434a-bcf3-1aed555bf91a_isrForCrosstalkSources_1125914_34.stderr parsl.dataflow.dflow INFO: Task 2334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c450c2d36a77fe4bb06444b2c3d08a1d with result from task 2334 parsl.dataflow.dflow DEBUG: Task 2334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/754b65a4-098e-4e2f-a784-f1e7c1e8a203_isrForCrosstalkSources_1125904_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/754b65a4-098e-4e2f-a784-f1e7c1e8a203_isrForCrosstalkSources_1125904_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:46 proc_lsst.multi:146 [INFO] found job 16896403 in provider astro proc_lsst.multi INFO: found job 16896403 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edd9406ec829b400c4def7f55258f943 with result from task 2336 parsl.dataflow.dflow DEBUG: Task 2336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/5b74e65a-b362-488e-9ce8-69d21d76cf12_isrForCrosstalkSources_1125889_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/5b74e65a-b362-488e-9ce8-69d21d76cf12_isrForCrosstalkSources_1125889_24.stderr parsl.dataflow.dflow INFO: Task 2362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 723661b05f8cf88ba45e3cb43bba5fbf with result from task 2362 parsl.dataflow.dflow DEBUG: Task 2362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/a0becab1-a20f-4a7f-804a-6b492771bfb3_isrForCrosstalkSources_1125941_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/a0becab1-a20f-4a7f-804a-6b492771bfb3_isrForCrosstalkSources_1125941_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:46 proc_lsst.multi:146 [INFO] found job 16896404 in provider astro proc_lsst.multi INFO: found job 16896404 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d26effb6c16717f1188a1035a7d6051 with result from task 2338 parsl.dataflow.dflow DEBUG: Task 2338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/c15ab890-5984-4b13-816e-f013c40afad6_isrForCrosstalkSources_1125899_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/c15ab890-5984-4b13-816e-f013c40afad6_isrForCrosstalkSources_1125899_24.stderr parsl.dataflow.dflow INFO: Task 2323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11a5fd2280be5a92faba621341a8b6f2 with result from task 2323 parsl.dataflow.dflow DEBUG: Task 2323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/96c6667f-be39-4bb8-b585-a6fbfbbaf5a0_isrForCrosstalkSources_1125941_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/96c6667f-be39-4bb8-b585-a6fbfbbaf5a0_isrForCrosstalkSources_1125941_7.stderr parsl.dataflow.dflow INFO: Task 2271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b53658c83fb15097f0e388ef159f2e52 with result from task 2271 parsl.dataflow.dflow DEBUG: Task 2271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/49028992-d6f3-47a9-bfdb-1f912f0435b9_isrForCrosstalkSources_1125917_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/49028992-d6f3-47a9-bfdb-1f912f0435b9_isrForCrosstalkSources_1125917_21.stderr parsl.dataflow.dflow INFO: Task 2345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 720f9e4650092d24b2bc69e232325071 with result from task 2345 parsl.dataflow.dflow DEBUG: Task 2345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e1ca6050-78c4-43f4-8b06-d5bcdaae93da_isrForCrosstalkSources_1125901_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e1ca6050-78c4-43f4-8b06-d5bcdaae93da_isrForCrosstalkSources_1125901_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:46 proc_lsst.multi:146 [INFO] found job 16896405 in provider astro proc_lsst.multi INFO: found job 16896405 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad6885fa3e33fb63c1d976a8a8a88318 with result from task 2333 parsl.dataflow.dflow DEBUG: Task 2333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/e388a90c-51a9-4960-8f62-3445337f285f_isrForCrosstalkSources_1125902_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/e388a90c-51a9-4960-8f62-3445337f285f_isrForCrosstalkSources_1125902_6.stderr parsl.dataflow.dflow INFO: Task 2252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cc504d0cb887396acafb09841e59fb9 with result from task 2252 parsl.dataflow.dflow DEBUG: Task 2252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0855394c-92ca-4142-a7c6-586093b468e3_isrForCrosstalkSources_1125914_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0855394c-92ca-4142-a7c6-586093b468e3_isrForCrosstalkSources_1125914_41.stderr parsl.dataflow.dflow INFO: Task 2407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe53ac664fec9711fa98289115a67a96 with result from task 2407 parsl.dataflow.dflow DEBUG: Task 2407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/a5c42aa5-e6cb-47d7-b5d0-96233b1b1555_isrForCrosstalkSources_1125915_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/a5c42aa5-e6cb-47d7-b5d0-96233b1b1555_isrForCrosstalkSources_1125915_18.stderr parsl.dataflow.dflow INFO: Task 2319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bcfa20f9d4786eb4477602ca44f4c29 with result from task 2319 parsl.dataflow.dflow DEBUG: Task 2319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/ecadd54d-f3ba-4ef8-a02a-1ae811ac9b83_isrForCrosstalkSources_1125891_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/ecadd54d-f3ba-4ef8-a02a-1ae811ac9b83_isrForCrosstalkSources_1125891_35.stderr parsl.dataflow.dflow INFO: Task 2418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a1f2eb778627dcd01af80960dba7cb3 with result from task 2418 parsl.dataflow.dflow DEBUG: Task 2418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/6a0682d9-8e94-4cc1-9d24-3bff04410e33_isrForCrosstalkSources_1125937_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/6a0682d9-8e94-4cc1-9d24-3bff04410e33_isrForCrosstalkSources_1125937_15.stderr parsl.dataflow.dflow INFO: Task 2344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc74dcaf3e79b614c72d55c6c048d467 with result from task 2344 parsl.dataflow.dflow DEBUG: Task 2344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/05724e60-bcfe-461a-84c1-ecb86def82da_isrForCrosstalkSources_1125940_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/05724e60-bcfe-461a-84c1-ecb86def82da_isrForCrosstalkSources_1125940_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:46 proc_lsst.multi:146 [INFO] found job 16896408 in provider astro proc_lsst.multi INFO: found job 16896408 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d23035436028266565a314db67c0ed6 with result from task 2360 parsl.dataflow.dflow DEBUG: Task 2360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/089e00e5-7564-469e-8d36-2a8baa86fedc_isrForCrosstalkSources_1125902_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/089e00e5-7564-469e-8d36-2a8baa86fedc_isrForCrosstalkSources_1125902_24.stderr parsl.dataflow.dflow INFO: Task 2395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 408760f35434a308b5ba2df20b52b147 with result from task 2395 parsl.dataflow.dflow DEBUG: Task 2395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/d213eebb-24e4-4ae4-aca3-772a2aa9bccd_isrForCrosstalkSources_1125889_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/d213eebb-24e4-4ae4-aca3-772a2aa9bccd_isrForCrosstalkSources_1125889_54.stderr parsl.dataflow.dflow INFO: Task 2351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e0a0d3201fe9ed75f4e3a846a01a689 with result from task 2351 parsl.dataflow.dflow DEBUG: Task 2351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/bff1be35-7859-470f-9ee6-54e06f95fb12_isrForCrosstalkSources_1125923_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/bff1be35-7859-470f-9ee6-54e06f95fb12_isrForCrosstalkSources_1125923_20.stderr parsl.dataflow.dflow INFO: Task 2353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ea90ab3603fc668b6d18878dffe6d99 with result from task 2353 parsl.dataflow.dflow DEBUG: Task 2353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/36c56643-b097-4c1c-8981-9f4846d572cc_isrForCrosstalkSources_1125891_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/36c56643-b097-4c1c-8981-9f4846d572cc_isrForCrosstalkSources_1125891_7.stderr parsl.dataflow.dflow INFO: Task 2368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d555cb00c7bb7ac85f24bbd294a7264 with result from task 2368 parsl.dataflow.dflow DEBUG: Task 2368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/2a3a5899-9a10-456b-a66c-92e3a451a87c_isrForCrosstalkSources_1125906_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/2a3a5899-9a10-456b-a66c-92e3a451a87c_isrForCrosstalkSources_1125906_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:47 proc_lsst.multi:146 [INFO] found job 16896409 in provider astro proc_lsst.multi INFO: found job 16896409 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91b87210069f399ae5e4773242be4847 with result from task 2318 parsl.dataflow.dflow DEBUG: Task 2318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/4e8744e5-2775-47e5-b90d-1df7d0fbb441_isrForCrosstalkSources_1125890_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/4e8744e5-2775-47e5-b90d-1df7d0fbb441_isrForCrosstalkSources_1125890_6.stderr parsl.dataflow.dflow INFO: Task 2393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5add9a3cd3c9fe3864e021eb01494fc with result from task 2393 parsl.dataflow.dflow DEBUG: Task 2393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/24b3c43b-8bb0-4338-ab2a-e267bf703a51_isrForCrosstalkSources_1125922_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/24b3c43b-8bb0-4338-ab2a-e267bf703a51_isrForCrosstalkSources_1125922_4.stderr parsl.dataflow.dflow INFO: Task 2383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df948c94156dfc5fc02a2289eccae242 with result from task 2383 parsl.dataflow.dflow DEBUG: Task 2383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/49477b2f-9cbf-4a6b-b03d-65d9936ac0a2_isrForCrosstalkSources_1125906_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/49477b2f-9cbf-4a6b-b03d-65d9936ac0a2_isrForCrosstalkSources_1125906_11.stderr parsl.dataflow.dflow INFO: Task 2428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 056703a3efbe0abf45227b8a548d4509 with result from task 2428 parsl.dataflow.dflow DEBUG: Task 2428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/61e35de7-0f86-4898-9b2c-bc86ccd7cf93_isrForCrosstalkSources_1125907_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/61e35de7-0f86-4898-9b2c-bc86ccd7cf93_isrForCrosstalkSources_1125907_32.stderr parsl.dataflow.dflow INFO: Task 2288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 632749d222163a826de807b8aa9c4bb5 with result from task 2288 parsl.dataflow.dflow DEBUG: Task 2288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/23e8b9fa-0dbe-421d-b044-8b9973c8224b_isrForCrosstalkSources_1125916_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/23e8b9fa-0dbe-421d-b044-8b9973c8224b_isrForCrosstalkSources_1125916_6.stderr parsl.dataflow.dflow INFO: Task 2374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea0d00b149bcce53a36bc54bd33252a0 with result from task 2374 parsl.dataflow.dflow DEBUG: Task 2374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/1082b762-d8a6-46ba-aade-3dac2b2cbe21_isrForCrosstalkSources_1125943_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/1082b762-d8a6-46ba-aade-3dac2b2cbe21_isrForCrosstalkSources_1125943_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:47 proc_lsst.multi:146 [INFO] found job 16896410 in provider astro proc_lsst.multi INFO: found job 16896410 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3cc4210e4c0b4db51a5e848e5d9ef70 with result from task 2429 parsl.dataflow.dflow DEBUG: Task 2429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c2c50ebe-d0cd-4476-b675-bbd918bb908c_isrForCrosstalkSources_1125904_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c2c50ebe-d0cd-4476-b675-bbd918bb908c_isrForCrosstalkSources_1125904_13.stderr parsl.dataflow.dflow INFO: Task 2346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b917a437a8ce9d4e5daba3ea85dcc293 with result from task 2346 parsl.dataflow.dflow DEBUG: Task 2346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/be5004ee-1740-452a-9a41-840ecfa42a35_isrForCrosstalkSources_1125891_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/be5004ee-1740-452a-9a41-840ecfa42a35_isrForCrosstalkSources_1125891_62.stderr parsl.dataflow.dflow INFO: Task 2380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcf0bf55d54903eaee48bf24a161c1b3 with result from task 2380 parsl.dataflow.dflow DEBUG: Task 2380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/7751bd4c-f060-4759-895b-f50ba239b923_isrForCrosstalkSources_1125940_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/7751bd4c-f060-4759-895b-f50ba239b923_isrForCrosstalkSources_1125940_18.stderr parsl.dataflow.dflow INFO: Task 2316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1a6e9ca7636ea326c8bc5ef0c475c88 with result from task 2316 parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:48 proc_lsst.multi:146 [INFO] found job 16896411 in provider astro proc_lsst.multi INFO: found job 16896411 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow DEBUG: Task 2316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/036a1450-0b25-4f8f-9a39-94b312cb3700_isrForCrosstalkSources_1125933_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/036a1450-0b25-4f8f-9a39-94b312cb3700_isrForCrosstalkSources_1125933_19.stderr parsl.dataflow.dflow INFO: Task 2372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe7c12eb986418d7fafc063648da7263 with result from task 2372 parsl.dataflow.dflow DEBUG: Task 2372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/7db01fab-b379-4243-958d-469eef5d01bb_isrForCrosstalkSources_1125892_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/7db01fab-b379-4243-958d-469eef5d01bb_isrForCrosstalkSources_1125892_2.stderr parsl.dataflow.dflow INFO: Task 2361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b1111b49ca75087403f70537c615f13 with result from task 2361 parsl.dataflow.dflow DEBUG: Task 2361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/745bf060-5328-46da-9d64-ea2a85091ce4_isrForCrosstalkSources_1125910_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/745bf060-5328-46da-9d64-ea2a85091ce4_isrForCrosstalkSources_1125910_43.stderr parsl.dataflow.dflow INFO: Task 2331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f6978e198ecfd4ce6d65007a87e3e62 with result from task 2331 parsl.dataflow.dflow DEBUG: Task 2331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/50937c70-402f-4143-a99e-2c4ec3a26969_isrForCrosstalkSources_1125909_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/50937c70-402f-4143-a99e-2c4ec3a26969_isrForCrosstalkSources_1125909_51.stderr parsl.dataflow.dflow INFO: Task 2432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd6cdf86422b2f9d1561f231309b32a9 with result from task 2432 parsl.dataflow.dflow DEBUG: Task 2432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/2179fed1-e274-44bd-959a-2d1dd2208e82_isrForCrosstalkSources_1125931_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/2179fed1-e274-44bd-959a-2d1dd2208e82_isrForCrosstalkSources_1125931_37.stderr parsl.dataflow.dflow INFO: Task 2356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83aa6f62d38baccd77b81c43cfcb3d57 with result from task 2356 parsl.dataflow.dflow DEBUG: Task 2356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/ac3d1199-1eb0-4179-8406-b06a0d9ac7cf_isrForCrosstalkSources_1125908_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/ac3d1199-1eb0-4179-8406-b06a0d9ac7cf_isrForCrosstalkSources_1125908_23.stderr parsl.dataflow.dflow INFO: Task 2283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09203cc98dd03e6afb8b74eb7d15d2ce with result from task 2283 parsl.dataflow.dflow DEBUG: Task 2283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/6c9b3c9f-3858-496f-9df4-9199cc7a87a1_isrForCrosstalkSources_1125942_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/6c9b3c9f-3858-496f-9df4-9199cc7a87a1_isrForCrosstalkSources_1125942_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:48 proc_lsst.multi:146 [INFO] found job 16896412 in provider astro proc_lsst.multi INFO: found job 16896412 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab63b077a292b4fe9a67780b0736ec55 with result from task 2348 parsl.dataflow.dflow DEBUG: Task 2348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/6e8bcc71-c2bf-44a0-ba70-71e99858ce9a_isrForCrosstalkSources_1125945_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/6e8bcc71-c2bf-44a0-ba70-71e99858ce9a_isrForCrosstalkSources_1125945_2.stderr parsl.dataflow.dflow INFO: Task 2354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57ef7382f448aa1fe911f079f4521dc8 with result from task 2354 parsl.dataflow.dflow DEBUG: Task 2354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/e83b3edf-346b-42d3-b0d2-f6ee8f469ce4_isrForCrosstalkSources_1125907_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/e83b3edf-346b-42d3-b0d2-f6ee8f469ce4_isrForCrosstalkSources_1125907_17.stderr parsl.dataflow.dflow INFO: Task 2350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb736f88c87b2606532adf473db0c7d2 with result from task 2350 parsl.dataflow.dflow DEBUG: Task 2350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/7ad9f680-e74b-43c9-803c-b9e22f5d1553_isrForCrosstalkSources_1125902_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/7ad9f680-e74b-43c9-803c-b9e22f5d1553_isrForCrosstalkSources_1125902_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:48 proc_lsst.multi:146 [INFO] found job 16896413 in provider astro proc_lsst.multi INFO: found job 16896413 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49048ec28eea52acdf8d6d51f04f59ba with result from task 2387 parsl.dataflow.dflow DEBUG: Task 2387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/caa117f1-d37a-42dc-8d56-d46c16451d02_isrForCrosstalkSources_1125916_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/caa117f1-d37a-42dc-8d56-d46c16451d02_isrForCrosstalkSources_1125916_10.stderr parsl.dataflow.dflow INFO: Task 2301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26e6de5ce77f40c2cd1bf04e70da0879 with result from task 2301 parsl.dataflow.dflow DEBUG: Task 2301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/74706dbc-797e-4173-b714-2bde268b47a6_isrForCrosstalkSources_1125894_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/74706dbc-797e-4173-b714-2bde268b47a6_isrForCrosstalkSources_1125894_59.stderr parsl.dataflow.dflow INFO: Task 2369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 435b95fe96073e3a4e4951ddeaf6a25e with result from task 2369 parsl.dataflow.dflow DEBUG: Task 2369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/fd3d7c56-1e34-4a20-9d68-2bf8208b229f_isrForCrosstalkSources_1125928_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/fd3d7c56-1e34-4a20-9d68-2bf8208b229f_isrForCrosstalkSources_1125928_12.stderr parsl.dataflow.dflow INFO: Task 2211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9acfc366b6fb08a429771580c49c7e86 with result from task 2211 parsl.dataflow.dflow DEBUG: Task 2211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/28f0a461-776a-43a5-9292-9f5167fe956e_isrForCrosstalkSources_1125930_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/28f0a461-776a-43a5-9292-9f5167fe956e_isrForCrosstalkSources_1125930_54.stderr parsl.dataflow.dflow INFO: Task 2180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df30189c5e9676a0e6601a6589f949be with result from task 2180 parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:48 proc_lsst.multi:146 [INFO] found job 16896414 in provider astro proc_lsst.multi INFO: found job 16896414 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow DEBUG: Task 2180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/6224a97a-8575-4ac9-abd5-4e7be9a6dbd3_isrForCrosstalkSources_1125911_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/6224a97a-8575-4ac9-abd5-4e7be9a6dbd3_isrForCrosstalkSources_1125911_4.stderr parsl.dataflow.dflow INFO: Task 2370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a65423e47c9c751eac4f30f36518179 with result from task 2370 parsl.dataflow.dflow DEBUG: Task 2370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/7a273fdb-b05f-4d55-a7f4-2b1bdc0e6924_isrForCrosstalkSources_1125930_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/7a273fdb-b05f-4d55-a7f4-2b1bdc0e6924_isrForCrosstalkSources_1125930_62.stderr parsl.dataflow.dflow INFO: Task 2285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88d545926b97bda0ec4806fdd7e9b256 with result from task 2285 parsl.dataflow.dflow DEBUG: Task 2285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3d1e8c57-56c5-4741-838e-1dde65a69921_isrForCrosstalkSources_1125912_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/3d1e8c57-56c5-4741-838e-1dde65a69921_isrForCrosstalkSources_1125912_10.stderr parsl.dataflow.dflow INFO: Task 2284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1abf2b5b15c55218d5fb0c3a836a21b9 with result from task 2284 parsl.dataflow.dflow DEBUG: Task 2284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/2b69771d-02ce-45cf-b00f-4dc8be7c44b1_isrForCrosstalkSources_1125936_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/2b69771d-02ce-45cf-b00f-4dc8be7c44b1_isrForCrosstalkSources_1125936_52.stderr parsl.dataflow.dflow INFO: Task 2443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50268e68bbc3f54d39d002651c38c98e with result from task 2443 parsl.dataflow.dflow DEBUG: Task 2443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/0f9bde07-8c59-4365-9a78-f0a99411396c_isrForCrosstalkSources_1125946_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/0f9bde07-8c59-4365-9a78-f0a99411396c_isrForCrosstalkSources_1125946_21.stderr parsl.dataflow.dflow INFO: Task 2458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef9e9bfb02bb40915be7699883c6c309 with result from task 2458 parsl.dataflow.dflow DEBUG: Task 2458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ed783c3e-1a36-40a7-b445-9a77f1db6f46_isrForCrosstalkSources_1125939_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ed783c3e-1a36-40a7-b445-9a77f1db6f46_isrForCrosstalkSources_1125939_15.stderr parsl.dataflow.dflow INFO: Task 2303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 598730f078ac1274ab514cb8fad117f4 with result from task 2303 parsl.dataflow.dflow DEBUG: Task 2303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/e23ef477-f5be-4b24-b12f-1aee4197da03_isrForCrosstalkSources_1125893_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/e23ef477-f5be-4b24-b12f-1aee4197da03_isrForCrosstalkSources_1125893_12.stderr parsl.dataflow.dflow INFO: Task 2311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a10e07311c2ca00fe2c316beeb6d9d8 with result from task 2311 parsl.dataflow.dflow DEBUG: Task 2311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/c9fb20c1-3f45-4c80-9907-ee0fb2788f3e_isrForCrosstalkSources_1125907_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/c9fb20c1-3f45-4c80-9907-ee0fb2788f3e_isrForCrosstalkSources_1125907_59.stderr parsl.dataflow.dflow INFO: Task 2256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 843bf13ecb4ae6dae420ed654f653e3b with result from task 2256 parsl.dataflow.dflow DEBUG: Task 2256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/e8369035-ccc8-496e-b5f7-744bdf6d6752_isrForCrosstalkSources_1125941_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/e8369035-ccc8-496e-b5f7-744bdf6d6752_isrForCrosstalkSources_1125941_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:49 proc_lsst.multi:146 [INFO] found job 16896415 in provider astro proc_lsst.multi INFO: found job 16896415 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac016deb3270351eddd80b36f62b99d6 with result from task 2367 parsl.dataflow.dflow DEBUG: Task 2367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/14c30d96-b98b-48bd-a910-bb27c998adf0_isrForCrosstalkSources_1125898_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/14c30d96-b98b-48bd-a910-bb27c998adf0_isrForCrosstalkSources_1125898_39.stderr parsl.dataflow.dflow INFO: Task 2365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6550e3a5ed89e613e8216df4ea9b62f with result from task 2365 parsl.dataflow.dflow DEBUG: Task 2365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f2c192b4-2bd9-4de7-b9e9-293c7af8e465_isrForCrosstalkSources_1125945_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f2c192b4-2bd9-4de7-b9e9-293c7af8e465_isrForCrosstalkSources_1125945_28.stderr parsl.dataflow.dflow INFO: Task 2434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10fc4ea1d6ff5677dff1be7e229d5c19 with result from task 2434 parsl.dataflow.dflow DEBUG: Task 2434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/13dd817d-4a2e-4392-8600-44d71e5b37e6_isrForCrosstalkSources_1125941_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/13dd817d-4a2e-4392-8600-44d71e5b37e6_isrForCrosstalkSources_1125941_51.stderr parsl.dataflow.dflow INFO: Task 2444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66eeec3a881a138997b7f3b9e444ea5b with result from task 2444 parsl.dataflow.dflow DEBUG: Task 2444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/68116c00-5445-4be7-8c37-4befcd909549_isrForCrosstalkSources_1125942_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/68116c00-5445-4be7-8c37-4befcd909549_isrForCrosstalkSources_1125942_56.stderr parsl.dataflow.dflow INFO: Task 2248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 887546980c32155d49215fd16fab5a29 with result from task 2248 parsl.dataflow.dflow DEBUG: Task 2248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/59e2a740-e6bf-41f6-8280-1a663bf1a327_isrForCrosstalkSources_1125934_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/59e2a740-e6bf-41f6-8280-1a663bf1a327_isrForCrosstalkSources_1125934_38.stderr parsl.dataflow.dflow INFO: Task 2371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25626d76dac4284fd46df325cc213a9b with result from task 2371 parsl.dataflow.dflow DEBUG: Task 2371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/f1bc8a0c-f011-4334-9efa-1751aa3dfa14_isrForCrosstalkSources_1125913_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/f1bc8a0c-f011-4334-9efa-1751aa3dfa14_isrForCrosstalkSources_1125913_7.stderr parsl.dataflow.dflow INFO: Task 2359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5d7f5f63781904e275679c8f0c61023 with result from task 2359 parsl.dataflow.dflow DEBUG: Task 2359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/09d9474e-db23-438f-91c8-06ec2ff43750_isrForCrosstalkSources_1125937_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/09d9474e-db23-438f-91c8-06ec2ff43750_isrForCrosstalkSources_1125937_52.stderr parsl.dataflow.dflow INFO: Task 2364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19ed3b4611078a4d36b7cc2ebee5f9cb with result from task 2364 parsl.dataflow.dflow DEBUG: Task 2364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/287336f5-40ee-42d8-91b4-8a5e1b9edd10_isrForCrosstalkSources_1125915_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/287336f5-40ee-42d8-91b4-8a5e1b9edd10_isrForCrosstalkSources_1125915_24.stderr parsl.dataflow.dflow INFO: Task 2329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4407086b1757bfa5056f05a06abd9c6d with result from task 2329 parsl.dataflow.dflow DEBUG: Task 2329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/0c1af82e-4d40-435b-ba52-6860027978b6_isrForCrosstalkSources_1125935_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/0c1af82e-4d40-435b-ba52-6860027978b6_isrForCrosstalkSources_1125935_3.stderr parsl.dataflow.dflow INFO: Task 2357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2ee181553cf1056b968713733368587 with result from task 2357 parsl.dataflow.dflow DEBUG: Task 2357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/dfbe6b3f-cd26-4c06-9523-0814f14cf361_isrForCrosstalkSources_1125942_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/dfbe6b3f-cd26-4c06-9523-0814f14cf361_isrForCrosstalkSources_1125942_21.stderr parsl.dataflow.dflow INFO: Task 2405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 981f44b7ecf30b406e83e4fbc51ef5e2 with result from task 2405 parsl.dataflow.dflow DEBUG: Task 2405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/59b9fc98-965b-45ed-9ec3-229f8914a4b9_isrForCrosstalkSources_1125940_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/59b9fc98-965b-45ed-9ec3-229f8914a4b9_isrForCrosstalkSources_1125940_19.stderr parsl.dataflow.dflow INFO: Task 2433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50a7fbd1a5c5bee3bceadf798aa7bb0d with result from task 2433 parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:49 proc_lsst.multi:146 [INFO] found job 16896416 in provider astro proc_lsst.multi INFO: found job 16896416 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow DEBUG: Task 2433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/30dcf23e-c1e8-44e4-9eba-428dfb098d49_isrForCrosstalkSources_1125889_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/30dcf23e-c1e8-44e4-9eba-428dfb098d49_isrForCrosstalkSources_1125889_3.stderr parsl.dataflow.dflow INFO: Task 2304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cebe5a84a0354b013080b0d53b065365 with result from task 2304 parsl.dataflow.dflow DEBUG: Task 2304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/a3bb0ce2-00f8-466f-bdb8-57710a757fbe_isrForCrosstalkSources_1125908_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/a3bb0ce2-00f8-466f-bdb8-57710a757fbe_isrForCrosstalkSources_1125908_42.stderr parsl.dataflow.dflow INFO: Task 2341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fcecdfe7d10f834aac31a73255ec571 with result from task 2341 parsl.dataflow.dflow DEBUG: Task 2341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/11afdd0a-9236-49d0-aaba-130362e42526_isrForCrosstalkSources_1125917_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/11afdd0a-9236-49d0-aaba-130362e42526_isrForCrosstalkSources_1125917_58.stderr parsl.dataflow.dflow INFO: Task 2276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c9dd799bb372007c654e8b6e0087c9c with result from task 2276 parsl.dataflow.dflow DEBUG: Task 2276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/dcdf2ef5-ca11-42f9-9d80-f5fbf438bb2d_isrForCrosstalkSources_1125922_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/dcdf2ef5-ca11-42f9-9d80-f5fbf438bb2d_isrForCrosstalkSources_1125922_49.stderr parsl.dataflow.dflow INFO: Task 2404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b0e5f556a2c048ac1256916088391fa with result from task 2404 parsl.dataflow.dflow DEBUG: Task 2404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/913b7f76-f746-4891-b442-9c7fc6e9af89_isrForCrosstalkSources_1125930_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/913b7f76-f746-4891-b442-9c7fc6e9af89_isrForCrosstalkSources_1125930_34.stderr parsl.dataflow.dflow INFO: Task 2175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cc28936462fa4b5871509ad408254fc with result from task 2175 parsl.dataflow.dflow DEBUG: Task 2175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/44963849-d5bf-4350-9f52-7fdf6121779b_isrForCrosstalkSources_1125922_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/44963849-d5bf-4350-9f52-7fdf6121779b_isrForCrosstalkSources_1125922_56.stderr parsl.dataflow.dflow INFO: Task 2440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae0277bd258dcff9d8f3329042190bda with result from task 2440 parsl.dataflow.dflow DEBUG: Task 2440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/656bab16-5634-4544-bd78-af2bfdbdea4e_isrForCrosstalkSources_1125932_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/656bab16-5634-4544-bd78-af2bfdbdea4e_isrForCrosstalkSources_1125932_51.stderr parsl.dataflow.dflow INFO: Task 2462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00fe17aefd01f59dd025e7eaf22c2aa7 with result from task 2462 parsl.dataflow.dflow DEBUG: Task 2462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/85c31f09-9c68-4411-83d9-0e03376a941f_isrForCrosstalkSources_1125890_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/85c31f09-9c68-4411-83d9-0e03376a941f_isrForCrosstalkSources_1125890_25.stderr parsl.dataflow.dflow INFO: Task 2435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry beb096b3e947f08a27493c2b4fce1b61 with result from task 2435 parsl.dataflow.dflow DEBUG: Task 2435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/b1efc0cf-cc24-4536-a48a-e2975cb6eaa4_isrForCrosstalkSources_1125901_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/b1efc0cf-cc24-4536-a48a-e2975cb6eaa4_isrForCrosstalkSources_1125901_10.stderr parsl.dataflow.dflow INFO: Task 2197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e5ff416260ef0adf55676ad5893f00f with result from task 2197 parsl.dataflow.dflow DEBUG: Task 2197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/dd823f63-7b89-451d-a99d-6dded3c04938_isrForCrosstalkSources_1125898_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/dd823f63-7b89-451d-a99d-6dded3c04938_isrForCrosstalkSources_1125898_46.stderr parsl.dataflow.dflow INFO: Task 2454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bc2e717e892a96f0bb05d8366fda5ef with result from task 2454 parsl.dataflow.dflow DEBUG: Task 2454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/8dde435c-ce5a-4ad5-8a5e-c2273392bde8_isrForCrosstalkSources_1125932_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/8dde435c-ce5a-4ad5-8a5e-c2273392bde8_isrForCrosstalkSources_1125932_32.stderr parsl.dataflow.dflow INFO: Task 2347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 941ff3e0a814acc58603d43b79268aad with result from task 2347 parsl.dataflow.dflow DEBUG: Task 2347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/65e723b5-1a5c-4906-8981-c2ae9c440b6c_isrForCrosstalkSources_1125897_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/65e723b5-1a5c-4906-8981-c2ae9c440b6c_isrForCrosstalkSources_1125897_10.stderr parsl.dataflow.dflow INFO: Task 2484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50cdca1eacc53d4e84f8f1e3d161142a with result from task 2484 parsl.dataflow.dflow DEBUG: Task 2484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/e85faa8e-dfa7-4f5a-beed-a20dbba2b9dd_isrForCrosstalkSources_1125927_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/e85faa8e-dfa7-4f5a-beed-a20dbba2b9dd_isrForCrosstalkSources_1125927_16.stderr parsl.dataflow.dflow INFO: Task 2377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95db143a43008c4c963b0f7869f53225 with result from task 2377 parsl.dataflow.dflow DEBUG: Task 2377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/96ea3005-ba8e-4b96-ab09-927e6791572d_isrForCrosstalkSources_1125946_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/96ea3005-ba8e-4b96-ab09-927e6791572d_isrForCrosstalkSources_1125946_17.stderr parsl.dataflow.dflow INFO: Task 2414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81f875ed614fe7e125a824bf64274c44 with result from task 2414 parsl.dataflow.dflow DEBUG: Task 2414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/c8837508-a55e-4c78-af08-fba903f55054_isrForCrosstalkSources_1125937_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/c8837508-a55e-4c78-af08-fba903f55054_isrForCrosstalkSources_1125937_20.stderr parsl.dataflow.dflow INFO: Task 2321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c13f884dab711e62d996afc8b8c33e4c with result from task 2321 parsl.dataflow.dflow DEBUG: Task 2321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/0af0a581-5828-433a-8e6b-d3f938eed9ac_isrForCrosstalkSources_1125894_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/0af0a581-5828-433a-8e6b-d3f938eed9ac_isrForCrosstalkSources_1125894_48.stderr parsl.dataflow.dflow INFO: Task 2396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49be3aba8e13a22b7c0a5a0e23ba1956 with result from task 2396 parsl.dataflow.dflow DEBUG: Task 2396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/9b994ebf-7934-4d8e-89fa-2f1f0380678b_isrForCrosstalkSources_1125927_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/9b994ebf-7934-4d8e-89fa-2f1f0380678b_isrForCrosstalkSources_1125927_23.stderr parsl.dataflow.dflow INFO: Task 2289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 582cd82deb6d68e1412e4b62e19ffb94 with result from task 2289 parsl.dataflow.dflow DEBUG: Task 2289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/618505d0-228f-4db3-b33c-11f9cf26058c_isrForCrosstalkSources_1125927_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/618505d0-228f-4db3-b33c-11f9cf26058c_isrForCrosstalkSources_1125927_43.stderr parsl.dataflow.dflow INFO: Task 2389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 640929ac3133f1b3f610b3695961d6ae with result from task 2389 parsl.dataflow.dflow DEBUG: Task 2389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d309e542-3e65-46d3-84d9-9c2a5f97bd4e_isrForCrosstalkSources_1125915_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d309e542-3e65-46d3-84d9-9c2a5f97bd4e_isrForCrosstalkSources_1125915_4.stderr parsl.dataflow.dflow INFO: Task 2436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b83199af4a11beeca2771f2cc4aa849b with result from task 2436 parsl.dataflow.dflow DEBUG: Task 2436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/2a39fa7d-a4a1-4bea-a510-b261c7dd9526_isrForCrosstalkSources_1125929_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/2a39fa7d-a4a1-4bea-a510-b261c7dd9526_isrForCrosstalkSources_1125929_18.stderr parsl.dataflow.dflow INFO: Task 2439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69487fcc8153bbb274ed90c90672c1d9 with result from task 2439 parsl.dataflow.dflow DEBUG: Task 2439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/c507a99d-1a95-4a21-89d4-3130bdef8350_isrForCrosstalkSources_1125929_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/c507a99d-1a95-4a21-89d4-3130bdef8350_isrForCrosstalkSources_1125929_17.stderr parsl.dataflow.dflow INFO: Task 2449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be5e2aad7a8e20d2030120f59e326dab with result from task 2449 parsl.dataflow.dflow DEBUG: Task 2449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/d4322d7e-10ee-4760-92d9-865e74a8d977_isrForCrosstalkSources_1125945_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/d4322d7e-10ee-4760-92d9-865e74a8d977_isrForCrosstalkSources_1125945_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:50 proc_lsst.multi:146 [INFO] found job 16896417 in provider astro proc_lsst.multi INFO: found job 16896417 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dc0d29b96fae4facb0b4b0474fe0505 with result from task 2375 parsl.dataflow.dflow DEBUG: Task 2375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/ddce7134-76f9-46b7-9d78-5e3c4b677e07_isrForCrosstalkSources_1125926_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/ddce7134-76f9-46b7-9d78-5e3c4b677e07_isrForCrosstalkSources_1125926_6.stderr parsl.dataflow.dflow INFO: Task 2413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e6e0a7cb92196ff21e963e649135c46 with result from task 2413 parsl.dataflow.dflow DEBUG: Task 2413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d6b689c5-4748-4065-8e37-23da7cefd0e7_isrForCrosstalkSources_1125890_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d6b689c5-4748-4065-8e37-23da7cefd0e7_isrForCrosstalkSources_1125890_62.stderr parsl.dataflow.dflow INFO: Task 2506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fccc38346554a19612100a226f95c15 with result from task 2506 parsl.dataflow.dflow DEBUG: Task 2506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/8939f7b3-1620-42bc-95ed-95fce42eff0d_isrForCrosstalkSources_1125900_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/8939f7b3-1620-42bc-95ed-95fce42eff0d_isrForCrosstalkSources_1125900_26.stderr parsl.dataflow.dflow INFO: Task 2170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 449a4e1de33b1342aa251578ab3f9ac9 with result from task 2170 parsl.dataflow.dflow DEBUG: Task 2170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/784ee28b-ad64-4bac-a7c5-5494126b7498_isrForCrosstalkSources_1125888_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/784ee28b-ad64-4bac-a7c5-5494126b7498_isrForCrosstalkSources_1125888_18.stderr parsl.dataflow.dflow INFO: Task 2465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3f747e608b76d32044a431fb7020c58 with result from task 2465 parsl.dataflow.dflow DEBUG: Task 2465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/72d1cd7c-ecff-4b90-b5a7-6c8de8d5b82e_isrForCrosstalkSources_1125894_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/72d1cd7c-ecff-4b90-b5a7-6c8de8d5b82e_isrForCrosstalkSources_1125894_8.stderr parsl.dataflow.dflow INFO: Task 2424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12785d6aa8b23cd29d3d9b5780d853ff with result from task 2424 parsl.dataflow.dflow DEBUG: Task 2424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/55c3ed71-9b70-4a86-98f9-fd62b1a3001c_isrForCrosstalkSources_1125933_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/55c3ed71-9b70-4a86-98f9-fd62b1a3001c_isrForCrosstalkSources_1125933_25.stderr parsl.dataflow.dflow INFO: Task 2208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5016ecaafa12d601912f76a45bd2425 with result from task 2208 parsl.dataflow.dflow DEBUG: Task 2208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/6dd00cd1-74f6-4954-84ab-854a84ce07d3_isrForCrosstalkSources_1125947_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/6dd00cd1-74f6-4954-84ab-854a84ce07d3_isrForCrosstalkSources_1125947_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:50 proc_lsst.multi:146 [INFO] found job 16896418 in provider astro proc_lsst.multi INFO: found job 16896418 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896385,16896386,16896387,16896388,16896389,16896390,16896392,16896394,16896395,16896396,16896397,16896398,16896399,16896400,16896401,16896402,16896403,16896404,16896405,16896408,16896409,16896410,16896411,16896412,16896413,16896414,16896415,16896416,16896417,16896418' parsl.dataflow.dflow INFO: Task 2417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21beb415a4d13889fd8178bb09f5d94d with result from task 2417 parsl.dataflow.dflow DEBUG: Task 2417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/b2d5eba5-2672-48bb-b0c7-ebd88afc6e4c_isrForCrosstalkSources_1125914_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/b2d5eba5-2672-48bb-b0c7-ebd88afc6e4c_isrForCrosstalkSources_1125914_25.stderr parsl.dataflow.dflow INFO: Task 2431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de302df3dab6fb2348d775e87bb97de4 with result from task 2431 parsl.dataflow.dflow DEBUG: Task 2431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/7c4712a3-29a1-4afa-8704-c19ca8583c83_isrForCrosstalkSources_1125898_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/7c4712a3-29a1-4afa-8704-c19ca8583c83_isrForCrosstalkSources_1125898_9.stderr parsl.dataflow.dflow INFO: Task 2442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dcea88e402d566eed834ed591a620d7 with result from task 2442 parsl.dataflow.dflow DEBUG: Task 2442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8dcbb2e5-b16d-4fdd-8420-3d952cb93589_isrForCrosstalkSources_1125944_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8dcbb2e5-b16d-4fdd-8420-3d952cb93589_isrForCrosstalkSources_1125944_20.stderr parsl.dataflow.dflow INFO: Task 2313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 975b0da3600fd79552389a53029ba7ab with result from task 2313 parsl.dataflow.dflow DEBUG: Task 2313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/5698ddf9-fa2a-4cd5-accd-fe9256e48cb4_isrForCrosstalkSources_1125936_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/5698ddf9-fa2a-4cd5-accd-fe9256e48cb4_isrForCrosstalkSources_1125936_29.stderr parsl.dataflow.dflow INFO: Task 2425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4815f7aeec0a87ff988d1ca829e14f62 with result from task 2425 parsl.dataflow.dflow DEBUG: Task 2425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/31c5e55c-0446-48ec-be2f-86e6864aacf7_isrForCrosstalkSources_1125923_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/31c5e55c-0446-48ec-be2f-86e6864aacf7_isrForCrosstalkSources_1125923_40.stderr parsl.dataflow.dflow INFO: Task 2388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed6303ca7f26f8e048233dca594d7b13 with result from task 2388 parsl.dataflow.dflow DEBUG: Task 2388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/bd0a3126-f39d-4043-ac1a-a8523f1b71d0_isrForCrosstalkSources_1125911_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/bd0a3126-f39d-4043-ac1a-a8523f1b71d0_isrForCrosstalkSources_1125911_37.stderr parsl.dataflow.dflow INFO: Task 2173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ab0f42fdc58404f2dcbe0dcc13b0656 with result from task 2173 parsl.dataflow.dflow DEBUG: Task 2173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/cb480439-3576-426c-ab23-73bf1b5c7e8b_isrForCrosstalkSources_1125928_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/cb480439-3576-426c-ab23-73bf1b5c7e8b_isrForCrosstalkSources_1125928_10.stderr parsl.dataflow.dflow INFO: Task 2381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef6342231bb5595ec31a9d8b35a4c116 with result from task 2381 parsl.dataflow.dflow DEBUG: Task 2381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/b781b13b-64cc-4d4b-b2c9-49ce209d0982_isrForCrosstalkSources_1125928_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/b781b13b-64cc-4d4b-b2c9-49ce209d0982_isrForCrosstalkSources_1125928_13.stderr parsl.dataflow.dflow INFO: Task 2390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34530994476b2272e52c09e96e0c710e with result from task 2390 parsl.dataflow.dflow DEBUG: Task 2390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/f7f5afd8-272c-47b4-99c1-4dfe5c0ed63d_isrForCrosstalkSources_1125892_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/f7f5afd8-272c-47b4-99c1-4dfe5c0ed63d_isrForCrosstalkSources_1125892_33.stderr parsl.dataflow.dflow INFO: Task 2363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bd9dc80e1bc325569936acfed84e551 with result from task 2363 parsl.dataflow.dflow DEBUG: Task 2363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/5793ea31-f41e-4903-893b-cb088be6aba5_isrForCrosstalkSources_1125910_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/5793ea31-f41e-4903-893b-cb088be6aba5_isrForCrosstalkSources_1125910_62.stderr parsl.dataflow.dflow INFO: Task 2457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c168671ad6d43913eaedceb521d854e5 with result from task 2457 parsl.dataflow.dflow DEBUG: Task 2457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/cbf3578d-a24e-42cd-942e-a11885e41e1d_isrForCrosstalkSources_1125932_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/cbf3578d-a24e-42cd-942e-a11885e41e1d_isrForCrosstalkSources_1125932_7.stderr parsl.dataflow.dflow INFO: Task 2199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b90e375a2f2fec4ef979dc137b052f7e with result from task 2199 parsl.dataflow.dflow DEBUG: Task 2199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/c8d975b6-7ab1-46be-9de7-1125ddebbc42_isrForCrosstalkSources_1125947_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/c8d975b6-7ab1-46be-9de7-1125ddebbc42_isrForCrosstalkSources_1125947_3.stderr parsl.dataflow.dflow INFO: Task 2513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c39822fb3a155a90d06b78a193a22fe9 with result from task 2513 parsl.dataflow.dflow DEBUG: Task 2513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/675e3a70-265d-4612-ba5b-676cd1cec474_isrForCrosstalkSources_1125941_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/675e3a70-265d-4612-ba5b-676cd1cec474_isrForCrosstalkSources_1125941_1.stderr parsl.dataflow.dflow INFO: Task 2420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5121a19fb809f074f8b6458ac2447ec8 with result from task 2420 parsl.dataflow.dflow DEBUG: Task 2420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/4db94df8-c123-4780-b2ca-3882969f5848_isrForCrosstalkSources_1125922_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/4db94df8-c123-4780-b2ca-3882969f5848_isrForCrosstalkSources_1125922_43.stderr parsl.dataflow.dflow INFO: Task 2451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4eab3409bbf50816346fca42a521f1e1 with result from task 2451 parsl.dataflow.dflow DEBUG: Task 2451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/138a2a5d-8859-42b4-ba2c-ea364ecc7f06_isrForCrosstalkSources_1125922_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/138a2a5d-8859-42b4-ba2c-ea364ecc7f06_isrForCrosstalkSources_1125922_38.stderr parsl.dataflow.dflow INFO: Task 2515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b14a6ef933c0dc20019817f50beb415 with result from task 2515 parsl.dataflow.dflow DEBUG: Task 2515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/240f66dc-9d21-4449-8198-a7e5950b92d5_isrForCrosstalkSources_1125915_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/240f66dc-9d21-4449-8198-a7e5950b92d5_isrForCrosstalkSources_1125915_13.stderr parsl.dataflow.dflow INFO: Task 2452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d8f58aadc698032a591ca6b9fca38c8 with result from task 2452 parsl.dataflow.dflow DEBUG: Task 2452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/40e6b5be-8149-48ca-b81a-b5eb35164e4c_isrForCrosstalkSources_1125942_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/40e6b5be-8149-48ca-b81a-b5eb35164e4c_isrForCrosstalkSources_1125942_34.stderr parsl.dataflow.dflow INFO: Task 2305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad4f2f83b37dcf032d5cf93e21fb1e0b with result from task 2305 parsl.dataflow.dflow DEBUG: Task 2305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/051867cb-935a-467a-81d2-d1021c011f92_isrForCrosstalkSources_1125932_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/051867cb-935a-467a-81d2-d1021c011f92_isrForCrosstalkSources_1125932_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896418 PD 16896417 PD 16896416 PD 16896415 PD 16896414 PD 16896413 PD 16896412 PD 16896411 PD 16896410 PD 16896409 PD 16896408 PD 16896405 PD 16896404 PD 16896403 PD 16896402 PD 16896401 PD 16896400 PD 16896399 PD 16896398 PD 16896397 PD 16896396 PD 16896395 PD 16896386 R 16896387 R 16896388 R 16896389 R 16896390 R 16896392 R 16896394 R 16896385 R parsl.providers.slurm.slurm DEBUG: Updating job 16896418 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896417 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896416 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896415 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896414 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896413 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896412 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896411 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896410 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896409 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896408 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896405 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896404 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896403 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896402 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896401 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896400 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896399 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896398 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896397 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896396 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896395 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896386 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896387 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896388 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896389 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896390 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896392 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896394 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896385 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:51 proc_lsst.multi:146 [INFO] found job 16896419 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896419 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.dataflow.dflow INFO: Task 2529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e4b1be10c94955dcea96e934d89200b with result from task 2529 parsl.dataflow.dflow DEBUG: Task 2529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/0636b521-2675-41e5-a2f0-dc00d1966614_isrForCrosstalkSources_1125910_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/0636b521-2675-41e5-a2f0-dc00d1966614_isrForCrosstalkSources_1125910_8.stderr parsl.dataflow.dflow INFO: Task 2275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff4fc90737b95a2749de8cf587696e4e with result from task 2275 parsl.dataflow.dflow DEBUG: Task 2275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/ea503690-4d94-4148-8ea5-5274a53e28b0_isrForCrosstalkSources_1125940_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/ea503690-4d94-4148-8ea5-5274a53e28b0_isrForCrosstalkSources_1125940_42.stderr parsl.dataflow.dflow INFO: Task 2463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80f6cb488c2330091d7b002eeb0ef63b with result from task 2463 parsl.dataflow.dflow DEBUG: Task 2463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/80a5249d-13be-48d3-b500-d8314d3c8e0f_isrForCrosstalkSources_1125889_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/80a5249d-13be-48d3-b500-d8314d3c8e0f_isrForCrosstalkSources_1125889_29.stderr parsl.dataflow.dflow INFO: Task 2461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94dda790a1779ca723ff5d61e9e17f5d with result from task 2461 parsl.dataflow.dflow DEBUG: Task 2461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4a9f2085-a480-454d-b1a8-618a9883ea4b_isrForCrosstalkSources_1125893_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4a9f2085-a480-454d-b1a8-618a9883ea4b_isrForCrosstalkSources_1125893_10.stderr parsl.dataflow.dflow INFO: Task 2430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a5d39bdf62719b0a861b14fe63f2f35 with result from task 2430 parsl.dataflow.dflow DEBUG: Task 2430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/644d53a7-2543-4613-9af6-ecd07f2d43bf_isrForCrosstalkSources_1125911_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/644d53a7-2543-4613-9af6-ecd07f2d43bf_isrForCrosstalkSources_1125911_29.stderr parsl.dataflow.dflow INFO: Task 2400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 353cc1c5162d131166f1753cbedd48c6 with result from task 2400 parsl.dataflow.dflow DEBUG: Task 2400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/80f6ff20-60d2-45c6-9796-2d16553465de_isrForCrosstalkSources_1125909_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/80f6ff20-60d2-45c6-9796-2d16553465de_isrForCrosstalkSources_1125909_45.stderr parsl.dataflow.dflow INFO: Task 2480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 816356be3711e951c73b98a3f4d86706 with result from task 2480 parsl.dataflow.dflow DEBUG: Task 2480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/3421f4f8-2776-40be-ad85-21e5f3568ee6_isrForCrosstalkSources_1125939_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/3421f4f8-2776-40be-ad85-21e5f3568ee6_isrForCrosstalkSources_1125939_8.stderr parsl.dataflow.dflow INFO: Task 2408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9980f1b93b9e62fceec571e86198f721 with result from task 2408 parsl.dataflow.dflow DEBUG: Task 2408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/ec6ac160-6477-43e5-9908-95f22bf5b4d8_isrForCrosstalkSources_1125928_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/ec6ac160-6477-43e5-9908-95f22bf5b4d8_isrForCrosstalkSources_1125928_58.stderr parsl.dataflow.dflow INFO: Task 2324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8302e894f779450cdc807bbdf22f79f5 with result from task 2324 parsl.dataflow.dflow DEBUG: Task 2324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/763eef04-181d-40b4-a784-feecb3efcdca_isrForCrosstalkSources_1125943_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/763eef04-181d-40b4-a784-feecb3efcdca_isrForCrosstalkSources_1125943_41.stderr parsl.dataflow.dflow INFO: Task 2392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcea90cc7011a1992835efd856bad60e with result from task 2392 parsl.dataflow.dflow DEBUG: Task 2392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c41bb072-b68e-4924-8a47-26ec51784acc_isrForCrosstalkSources_1125917_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c41bb072-b68e-4924-8a47-26ec51784acc_isrForCrosstalkSources_1125917_7.stderr parsl.dataflow.dflow INFO: Task 2278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 207dfc4f957e73b3ac4a6e77d974417d with result from task 2278 parsl.dataflow.dflow DEBUG: Task 2278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/133f2afd-c1dc-45db-858f-74696f0c8453_isrForCrosstalkSources_1125941_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/133f2afd-c1dc-45db-858f-74696f0c8453_isrForCrosstalkSources_1125941_30.stderr parsl.dataflow.dflow INFO: Task 2527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdd1339e66954c942ee330095573bd9f with result from task 2527 parsl.dataflow.dflow DEBUG: Task 2527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/f423c108-ab9e-468c-81d7-e2f46737a09f_isrForCrosstalkSources_1125920_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/f423c108-ab9e-468c-81d7-e2f46737a09f_isrForCrosstalkSources_1125920_26.stderr parsl.dataflow.dflow INFO: Task 2486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2cc94cdbf825fd99065ae8c7e85b754 with result from task 2486 parsl.dataflow.dflow DEBUG: Task 2486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d8cd2914-93e0-45e7-acf2-d24a5736a57d_isrForCrosstalkSources_1125890_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d8cd2914-93e0-45e7-acf2-d24a5736a57d_isrForCrosstalkSources_1125890_51.stderr parsl.dataflow.dflow INFO: Task 2282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c95325c851d2a22232b40f0f35450116 with result from task 2282 parsl.dataflow.dflow DEBUG: Task 2282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/501cd0f7-d786-45c1-83c5-9c518e53e517_isrForCrosstalkSources_1125903_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/501cd0f7-d786-45c1-83c5-9c518e53e517_isrForCrosstalkSources_1125903_53.stderr parsl.dataflow.dflow INFO: Task 2416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8e6c5036c17ba8957358e3de4293042 with result from task 2416 parsl.dataflow.dflow DEBUG: Task 2416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/9850f5ad-23f8-4e6a-b6ce-63c6f7db00d5_isrForCrosstalkSources_1125902_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/9850f5ad-23f8-4e6a-b6ce-63c6f7db00d5_isrForCrosstalkSources_1125902_18.stderr parsl.dataflow.dflow INFO: Task 2378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 429c24792b99ace5f2f4515230f64a03 with result from task 2378 parsl.dataflow.dflow DEBUG: Task 2378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/f17add48-ac71-4637-b36f-694020352212_isrForCrosstalkSources_1125906_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/f17add48-ac71-4637-b36f-694020352212_isrForCrosstalkSources_1125906_46.stderr parsl.dataflow.dflow INFO: Task 2477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2319304926720424c76c251179c6275b with result from task 2477 parsl.dataflow.dflow DEBUG: Task 2477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/ff1b01b2-7290-4974-b337-ce7c07be68fa_isrForCrosstalkSources_1125935_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/ff1b01b2-7290-4974-b337-ce7c07be68fa_isrForCrosstalkSources_1125935_60.stderr parsl.dataflow.dflow INFO: Task 2401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e10dcd5d98766209f393508cbc67ad68 with result from task 2401 parsl.dataflow.dflow DEBUG: Task 2401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/500e4ad3-d1f0-4cbb-97d6-18d4ab81ae43_isrForCrosstalkSources_1125899_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/500e4ad3-d1f0-4cbb-97d6-18d4ab81ae43_isrForCrosstalkSources_1125899_34.stderr parsl.dataflow.dflow INFO: Task 2519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d04ec33eb1d7b786fcef3d4a90852295 with result from task 2519 parsl.dataflow.dflow DEBUG: Task 2519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/e597966d-4e39-443f-8a21-124bab1cf2ff_isrForCrosstalkSources_1125906_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/e597966d-4e39-443f-8a21-124bab1cf2ff_isrForCrosstalkSources_1125906_34.stderr parsl.dataflow.dflow INFO: Task 2446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99e8ac8df428c9b99d86e23372bdacc9 with result from task 2446 parsl.dataflow.dflow DEBUG: Task 2446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6d994576-c3f5-40f1-a7a3-3c9b9b73ecfd_isrForCrosstalkSources_1125909_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6d994576-c3f5-40f1-a7a3-3c9b9b73ecfd_isrForCrosstalkSources_1125909_7.stderr parsl.dataflow.dflow INFO: Task 2533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 080754a28f9723dee49df6e5a129c3b4 with result from task 2533 parsl.dataflow.dflow DEBUG: Task 2533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/7ebce183-f6fa-41b0-b92a-ca0d327e87a0_isrForCrosstalkSources_1125893_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/7ebce183-f6fa-41b0-b92a-ca0d327e87a0_isrForCrosstalkSources_1125893_26.stderr parsl.dataflow.dflow INFO: Task 2471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f507c14a2a9069092b0c8cf7200a5978 with result from task 2471 parsl.dataflow.dflow DEBUG: Task 2471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/f75b4542-6f9d-4bdc-addf-7566ae78b5b3_isrForCrosstalkSources_1125912_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/f75b4542-6f9d-4bdc-addf-7566ae78b5b3_isrForCrosstalkSources_1125912_34.stderr parsl.dataflow.dflow INFO: Task 2493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 382c96736b3512284c72cc30e2ba82db with result from task 2493 parsl.dataflow.dflow DEBUG: Task 2493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/e3b96591-6895-45ca-bada-8adc9e6ee66e_isrForCrosstalkSources_1125897_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/e3b96591-6895-45ca-bada-8adc9e6ee66e_isrForCrosstalkSources_1125897_26.stderr parsl.dataflow.dflow INFO: Task 2524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38d6184914f703cf1ee5222aa0b2ef7c with result from task 2524 parsl.dataflow.dflow DEBUG: Task 2524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/88d75cd0-1818-4752-82ba-515746641a85_isrForCrosstalkSources_1125925_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/88d75cd0-1818-4752-82ba-515746641a85_isrForCrosstalkSources_1125925_26.stderr parsl.dataflow.dflow INFO: Task 2479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9668a86837740fd9a1a26b1b38685652 with result from task 2479 parsl.dataflow.dflow DEBUG: Task 2479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/87c7c04c-9438-48c0-aced-a265ede3cec0_isrForCrosstalkSources_1125944_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/87c7c04c-9438-48c0-aced-a265ede3cec0_isrForCrosstalkSources_1125944_32.stderr parsl.dataflow.dflow INFO: Task 2332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccb6ce120800c911dfb570cea3989c7b with result from task 2332 parsl.dataflow.dflow DEBUG: Task 2332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/c84a730a-75d2-4019-99f1-fcf42cde329d_isrForCrosstalkSources_1125920_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/c84a730a-75d2-4019-99f1-fcf42cde329d_isrForCrosstalkSources_1125920_31.stderr parsl.dataflow.dflow INFO: Task 2422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e64d31c095709414aa6b91cf0029df8 with result from task 2422 parsl.dataflow.dflow DEBUG: Task 2422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/61801768-da6d-4061-8755-95f30d99f13d_isrForCrosstalkSources_1125905_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/61801768-da6d-4061-8755-95f30d99f13d_isrForCrosstalkSources_1125905_21.stderr parsl.dataflow.dflow INFO: Task 2366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c55f4b1176916e57ce87d7965f7aabff with result from task 2366 parsl.dataflow.dflow DEBUG: Task 2366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/5976ce11-c87f-4e1a-8b0e-70c46075e2f5_isrForCrosstalkSources_1125938_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/5976ce11-c87f-4e1a-8b0e-70c46075e2f5_isrForCrosstalkSources_1125938_55.stderr parsl.dataflow.dflow INFO: Task 2512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbf83f0bbe668647cb7f1db179776613 with result from task 2512 parsl.dataflow.dflow DEBUG: Task 2512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/dbcc0264-c4ab-4757-823c-528f492ada2f_isrForCrosstalkSources_1125891_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/dbcc0264-c4ab-4757-823c-528f492ada2f_isrForCrosstalkSources_1125891_30.stderr parsl.dataflow.dflow INFO: Task 2391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89475cbfc230825c6457bf330699cc91 with result from task 2391 parsl.dataflow.dflow DEBUG: Task 2391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e0f0b5aa-4cf9-4a8e-9e92-946863c27f72_isrForCrosstalkSources_1125942_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e0f0b5aa-4cf9-4a8e-9e92-946863c27f72_isrForCrosstalkSources_1125942_61.stderr parsl.dataflow.dflow INFO: Task 2386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83f860a53a19d8cef59099eb70dfc14d with result from task 2386 parsl.dataflow.dflow DEBUG: Task 2386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/d32ae83d-6c16-4299-980a-7dcf67e83f15_isrForCrosstalkSources_1125938_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/d32ae83d-6c16-4299-980a-7dcf67e83f15_isrForCrosstalkSources_1125938_53.stderr parsl.dataflow.dflow INFO: Task 2409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73511e08db5cc72f907031bbcc1a4cfc with result from task 2409 parsl.dataflow.dflow DEBUG: Task 2409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/bf9bac0c-57ef-4348-b470-e5d6691f39c5_isrForCrosstalkSources_1125947_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/bf9bac0c-57ef-4348-b470-e5d6691f39c5_isrForCrosstalkSources_1125947_50.stderr parsl.dataflow.dflow INFO: Task 2504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12048b8dae777dc04bd1607105a9a8c0 with result from task 2504 parsl.dataflow.dflow DEBUG: Task 2504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0de1c688-a147-423c-a4d3-bc32fe380326_isrForCrosstalkSources_1125904_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/0de1c688-a147-423c-a4d3-bc32fe380326_isrForCrosstalkSources_1125904_33.stderr parsl.dataflow.dflow INFO: Task 2498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2c714762a10db2ef1f2fe83b868affc with result from task 2498 parsl.dataflow.dflow DEBUG: Task 2498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/6e7bc300-70af-4e4a-aee9-b60fe9cdc294_isrForCrosstalkSources_1125911_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/6e7bc300-70af-4e4a-aee9-b60fe9cdc294_isrForCrosstalkSources_1125911_49.stderr parsl.dataflow.dflow INFO: Task 2411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25ef6985b952e81f0e64ed1cf45a83a6 with result from task 2411 parsl.dataflow.dflow DEBUG: Task 2411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/d10563d7-b901-496d-8af9-51da672708a7_isrForCrosstalkSources_1125945_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/d10563d7-b901-496d-8af9-51da672708a7_isrForCrosstalkSources_1125945_35.stderr parsl.dataflow.dflow INFO: Task 2373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c7252609c87ae19741e15b88797d042 with result from task 2373 parsl.dataflow.dflow DEBUG: Task 2373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/fc99383e-006a-4556-8eb5-196ef8b7e906_isrForCrosstalkSources_1125895_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/fc99383e-006a-4556-8eb5-196ef8b7e906_isrForCrosstalkSources_1125895_52.stderr parsl.dataflow.dflow INFO: Task 2534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 983e2f6ec0af55f8599cb40e1c3b3fc7 with result from task 2534 parsl.dataflow.dflow DEBUG: Task 2534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/6253ff47-8005-4e2a-9c10-845996c683dc_isrForCrosstalkSources_1125906_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/6253ff47-8005-4e2a-9c10-845996c683dc_isrForCrosstalkSources_1125906_22.stderr parsl.dataflow.dflow INFO: Task 2402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddaabc7994ca0ba5eaf526ba9b65689d with result from task 2402 parsl.dataflow.dflow DEBUG: Task 2402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/21d840d1-6ffd-493a-9ae1-dd97e51c2a59_isrForCrosstalkSources_1125911_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/21d840d1-6ffd-493a-9ae1-dd97e51c2a59_isrForCrosstalkSources_1125911_6.stderr parsl.dataflow.dflow INFO: Task 2397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 612c3157b225dcc16c032da5e9406c4f with result from task 2397 parsl.dataflow.dflow DEBUG: Task 2397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/52866641-a0c9-431d-9bf6-02d2f9649b27_isrForCrosstalkSources_1125913_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/52866641-a0c9-431d-9bf6-02d2f9649b27_isrForCrosstalkSources_1125913_9.stderr parsl.dataflow.dflow INFO: Task 2500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cb07de37592e25e618d5fa3b6bbda24 with result from task 2500 parsl.dataflow.dflow DEBUG: Task 2500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/d361014e-8336-4fef-8e86-d484eee2355a_isrForCrosstalkSources_1125930_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/d361014e-8336-4fef-8e86-d484eee2355a_isrForCrosstalkSources_1125930_38.stderr parsl.dataflow.dflow INFO: Task 2437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 305109f2d53672d683b99cbc6582acb6 with result from task 2437 parsl.dataflow.dflow DEBUG: Task 2437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/13a3eb5a-f8bc-407e-9ae6-6ecb1a8eb08c_isrForCrosstalkSources_1125904_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/13a3eb5a-f8bc-407e-9ae6-6ecb1a8eb08c_isrForCrosstalkSources_1125904_2.stderr parsl.dataflow.dflow INFO: Task 2427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3143c03f291f3c35d5bdbd568e2dc7c2 with result from task 2427 parsl.dataflow.dflow DEBUG: Task 2427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/31b23d9a-77c8-425e-b29c-fbc3dd77a16a_isrForCrosstalkSources_1125908_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/31b23d9a-77c8-425e-b29c-fbc3dd77a16a_isrForCrosstalkSources_1125908_33.stderr parsl.dataflow.dflow INFO: Task 2217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23309571569f5e3c2d61727a5c34ca67 with result from task 2217 parsl.dataflow.dflow DEBUG: Task 2217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/8e55cff0-9911-4d5b-aa61-31f77986a7ae_isrForCrosstalkSources_1125916_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/8e55cff0-9911-4d5b-aa61-31f77986a7ae_isrForCrosstalkSources_1125916_17.stderr parsl.dataflow.dflow INFO: Task 2231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea569f91cda12fff8999ff05a2ac37d6 with result from task 2231 parsl.dataflow.dflow DEBUG: Task 2231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/0dd5f3d7-e720-4446-a24d-f2eab4b65090_isrForCrosstalkSources_1125931_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/0dd5f3d7-e720-4446-a24d-f2eab4b65090_isrForCrosstalkSources_1125931_3.stderr parsl.dataflow.dflow INFO: Task 2419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b035e82bd5cf3e1a48c8f318e1c83bec with result from task 2419 parsl.dataflow.dflow DEBUG: Task 2419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/f479768f-27e7-4c49-b2a4-2b9b187411d2_isrForCrosstalkSources_1125895_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/f479768f-27e7-4c49-b2a4-2b9b187411d2_isrForCrosstalkSources_1125895_44.stderr parsl.dataflow.dflow INFO: Task 2483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2897edd6d2849813ce0e3b7145b4470 with result from task 2483 parsl.dataflow.dflow DEBUG: Task 2483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/1241e008-385a-48c3-b7a8-dbd6bd4deecb_isrForCrosstalkSources_1125892_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/1241e008-385a-48c3-b7a8-dbd6bd4deecb_isrForCrosstalkSources_1125892_40.stderr parsl.dataflow.dflow INFO: Task 2490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71d4d200a1eb6754b2f7d0c7ef9d5457 with result from task 2490 parsl.dataflow.dflow DEBUG: Task 2490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/47fb2d24-b45f-4418-8b3a-da929e74f41e_isrForCrosstalkSources_1125914_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/47fb2d24-b45f-4418-8b3a-da929e74f41e_isrForCrosstalkSources_1125914_51.stderr parsl.dataflow.dflow INFO: Task 2467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64874f529429b439a3acca933fe2e0b4 with result from task 2467 parsl.dataflow.dflow DEBUG: Task 2467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/e5a8e964-8a6f-41c4-b3d6-d0b6c184e673_isrForCrosstalkSources_1125889_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/e5a8e964-8a6f-41c4-b3d6-d0b6c184e673_isrForCrosstalkSources_1125889_25.stderr parsl.dataflow.dflow INFO: Task 2518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7444808bb7f52066ea7fac182cac311c with result from task 2518 parsl.dataflow.dflow DEBUG: Task 2518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/0d0f41a8-406e-4931-ad45-cf5ff23eab9f_isrForCrosstalkSources_1125926_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/0d0f41a8-406e-4931-ad45-cf5ff23eab9f_isrForCrosstalkSources_1125926_40.stderr parsl.dataflow.dflow INFO: Task 2488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b611300201ae3f7818d46cbe4d30eb0a with result from task 2488 parsl.dataflow.dflow DEBUG: Task 2488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/04527293-7420-44c4-a9d9-5d8842922ecd_isrForCrosstalkSources_1125889_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/04527293-7420-44c4-a9d9-5d8842922ecd_isrForCrosstalkSources_1125889_56.stderr parsl.dataflow.dflow INFO: Task 2423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89b18c47c049a29328dcb138e9c4412e with result from task 2423 parsl.dataflow.dflow DEBUG: Task 2423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/905a2c2b-4516-4b9e-9af3-cd658f35974a_isrForCrosstalkSources_1125941_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/905a2c2b-4516-4b9e-9af3-cd658f35974a_isrForCrosstalkSources_1125941_3.stderr parsl.dataflow.dflow INFO: Task 2537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdd3b2816d8ace0a992aeb4b7ced6b28 with result from task 2537 parsl.dataflow.dflow DEBUG: Task 2537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/24d3a7ed-0b0b-436c-8111-658784c0f9e1_isrForCrosstalkSources_1125932_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/24d3a7ed-0b0b-436c-8111-658784c0f9e1_isrForCrosstalkSources_1125932_22.stderr parsl.dataflow.dflow INFO: Task 2478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 303e6c98a77be2aec0daebcd6ea5a694 with result from task 2478 parsl.dataflow.dflow DEBUG: Task 2478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/2db39b21-a874-4675-824e-d3b4beb3e12e_isrForCrosstalkSources_1125936_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/2db39b21-a874-4675-824e-d3b4beb3e12e_isrForCrosstalkSources_1125936_13.stderr parsl.dataflow.dflow INFO: Task 2472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ad39e1aa6c5962cf1e75c5fe0abc4ae with result from task 2472 parsl.dataflow.dflow DEBUG: Task 2472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/4c14bdd5-f22d-4090-8db6-c9eeba7ef03c_isrForCrosstalkSources_1125889_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/4c14bdd5-f22d-4090-8db6-c9eeba7ef03c_isrForCrosstalkSources_1125889_37.stderr parsl.dataflow.dflow INFO: Task 2456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26591f6e3ee38e76fed9eba540509b26 with result from task 2456 parsl.dataflow.dflow DEBUG: Task 2456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/dff1f8be-3913-435c-b625-979b5259599e_isrForCrosstalkSources_1125939_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/dff1f8be-3913-435c-b625-979b5259599e_isrForCrosstalkSources_1125939_34.stderr parsl.dataflow.dflow INFO: Task 2410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cb6927575a7533663ab52940b19b3b0 with result from task 2410 parsl.dataflow.dflow DEBUG: Task 2410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/6fbf9765-cc24-459d-b804-10b8f6a7e738_isrForCrosstalkSources_1125924_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/6fbf9765-cc24-459d-b804-10b8f6a7e738_isrForCrosstalkSources_1125924_51.stderr parsl.dataflow.dflow INFO: Task 2510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa15ce2b4902c5192c97e85012ab25fe with result from task 2510 parsl.dataflow.dflow DEBUG: Task 2510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/7412d0d0-c5d8-48d9-9ba7-8f4363baae36_isrForCrosstalkSources_1125889_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/7412d0d0-c5d8-48d9-9ba7-8f4363baae36_isrForCrosstalkSources_1125889_61.stderr parsl.dataflow.dflow INFO: Task 2505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1765a04a27cc6cbfa669dc32dbcda53 with result from task 2505 parsl.dataflow.dflow DEBUG: Task 2505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/ae2daec1-8f9a-4707-9860-01b230e15cab_isrForCrosstalkSources_1125905_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/ae2daec1-8f9a-4707-9860-01b230e15cab_isrForCrosstalkSources_1125905_18.stderr parsl.dataflow.dflow INFO: Task 2447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d34b729f3107dd4ff5beab7c9bd8c56e with result from task 2447 parsl.dataflow.dflow DEBUG: Task 2447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/69b77c70-64fb-43e5-8d3c-8af96ea15561_isrForCrosstalkSources_1125929_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/69b77c70-64fb-43e5-8d3c-8af96ea15561_isrForCrosstalkSources_1125929_10.stderr parsl.dataflow.dflow INFO: Task 2464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68abebd49d8b914b4056b1e683a950a6 with result from task 2464 parsl.dataflow.dflow DEBUG: Task 2464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/a9f0e556-8a48-4209-b486-949109374457_isrForCrosstalkSources_1125917_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/a9f0e556-8a48-4209-b486-949109374457_isrForCrosstalkSources_1125917_4.stderr parsl.dataflow.dflow INFO: Task 2453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc1a536cb90eefebd15e995391fcf538 with result from task 2453 parsl.dataflow.dflow DEBUG: Task 2453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/064e9b0c-9810-4d7b-bc3e-dc1b039068d5_isrForCrosstalkSources_1125927_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/064e9b0c-9810-4d7b-bc3e-dc1b039068d5_isrForCrosstalkSources_1125927_7.stderr parsl.dataflow.dflow INFO: Task 2224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77f390d3a82f83097a36625ef6670db8 with result from task 2224 parsl.dataflow.dflow DEBUG: Task 2224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/fcc4b907-e812-48b4-b96f-c81727382cd6_isrForCrosstalkSources_1125931_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/fcc4b907-e812-48b4-b96f-c81727382cd6_isrForCrosstalkSources_1125931_60.stderr parsl.dataflow.dflow INFO: Task 2459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 226b38927570eed28b9a18b46aa8eda8 with result from task 2459 parsl.dataflow.dflow DEBUG: Task 2459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/f7816967-728e-4127-85ed-e1cfa9278b48_isrForCrosstalkSources_1125891_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/f7816967-728e-4127-85ed-e1cfa9278b48_isrForCrosstalkSources_1125891_36.stderr parsl.dataflow.dflow INFO: Task 2496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18f4e926ae405a8ead997ec11529bb5e with result from task 2496 parsl.dataflow.dflow DEBUG: Task 2496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/9335968f-b7ae-45c6-863f-414fb21106aa_isrForCrosstalkSources_1125929_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/9335968f-b7ae-45c6-863f-414fb21106aa_isrForCrosstalkSources_1125929_28.stderr parsl.dataflow.dflow INFO: Task 2514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f4025668ad4b2228496e9f86228f877 with result from task 2514 parsl.dataflow.dflow DEBUG: Task 2514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/761ac2b9-8002-48fb-8773-c94aa4794d2c_isrForCrosstalkSources_1125902_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/761ac2b9-8002-48fb-8773-c94aa4794d2c_isrForCrosstalkSources_1125902_11.stderr parsl.dataflow.dflow INFO: Task 2494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f269550dc612086ebff72a01ce65068e with result from task 2494 parsl.dataflow.dflow DEBUG: Task 2494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/1774d8b4-a0ce-47c3-8b5e-84643d3fd6dc_isrForCrosstalkSources_1125930_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/1774d8b4-a0ce-47c3-8b5e-84643d3fd6dc_isrForCrosstalkSources_1125930_42.stderr parsl.dataflow.dflow INFO: Task 2525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5607c80bf85fff5f219b144e87e2fe9 with result from task 2525 parsl.dataflow.dflow DEBUG: Task 2525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/afb24515-d6cc-4f7e-a44a-42f37240191a_isrForCrosstalkSources_1125933_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/afb24515-d6cc-4f7e-a44a-42f37240191a_isrForCrosstalkSources_1125933_7.stderr parsl.dataflow.dflow INFO: Task 2421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9229660356e062ede3ac1ac5ab386442 with result from task 2421 parsl.dataflow.dflow DEBUG: Task 2421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/28c405d2-a20c-4601-a73b-de68824c444b_isrForCrosstalkSources_1125904_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/28c405d2-a20c-4601-a73b-de68824c444b_isrForCrosstalkSources_1125904_49.stderr parsl.dataflow.dflow INFO: Task 2475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68357bad0601cf757f8f14729ab02b11 with result from task 2475 parsl.dataflow.dflow DEBUG: Task 2475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/cec494d1-b7cd-4cf2-bc6e-984830856d10_isrForCrosstalkSources_1125905_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/cec494d1-b7cd-4cf2-bc6e-984830856d10_isrForCrosstalkSources_1125905_12.stderr parsl.dataflow.dflow INFO: Task 2474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0daa5119198c1cf5c5ef121721b432ca with result from task 2474 parsl.dataflow.dflow DEBUG: Task 2474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/78a93111-6b11-43ee-b465-b2b6c2441546_isrForCrosstalkSources_1125905_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/78a93111-6b11-43ee-b465-b2b6c2441546_isrForCrosstalkSources_1125905_52.stderr parsl.dataflow.dflow INFO: Task 2438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d949eb109951d26b3a4cecdac629ceb3 with result from task 2438 parsl.dataflow.dflow DEBUG: Task 2438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/21067d40-d76f-4c02-b340-b2baea0a3470_isrForCrosstalkSources_1125937_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/21067d40-d76f-4c02-b340-b2baea0a3470_isrForCrosstalkSources_1125937_3.stderr parsl.dataflow.dflow INFO: Task 2497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ada9eb2c40ecf38b75a9702b520bcfe with result from task 2497 parsl.dataflow.dflow DEBUG: Task 2497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/2b91d6f6-fa57-4371-92e8-83186d90b96c_isrForCrosstalkSources_1125932_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/2b91d6f6-fa57-4371-92e8-83186d90b96c_isrForCrosstalkSources_1125932_3.stderr parsl.dataflow.dflow INFO: Task 2523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ac866ce06e35782ed389ae53449fe18 with result from task 2523 parsl.dataflow.dflow DEBUG: Task 2523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/ab553435-02a2-49ff-a6d1-827030ae8512_isrForCrosstalkSources_1125945_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/ab553435-02a2-49ff-a6d1-827030ae8512_isrForCrosstalkSources_1125945_6.stderr parsl.dataflow.dflow INFO: Task 2503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df0fb6214d7bd519bb1f8e124ae5d117 with result from task 2503 parsl.dataflow.dflow DEBUG: Task 2503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/147b0419-a7db-4bb4-b56d-a6f85210b764_isrForCrosstalkSources_1125945_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/147b0419-a7db-4bb4-b56d-a6f85210b764_isrForCrosstalkSources_1125945_36.stderr parsl.dataflow.dflow INFO: Task 2460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d7ca8ede17d62aa08bd075f52c5a598 with result from task 2460 parsl.dataflow.dflow DEBUG: Task 2460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f684dfb3-8576-489e-89d3-36026ea34f94_isrForCrosstalkSources_1125894_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/f684dfb3-8576-489e-89d3-36026ea34f94_isrForCrosstalkSources_1125894_60.stderr parsl.dataflow.dflow INFO: Task 2161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46597bdcf78bf47e8c79e4abad3acfa1 with result from task 2161 parsl.dataflow.dflow DEBUG: Task 2161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/420767d4-810c-420a-a189-dc37b1096196_isrForCrosstalkSources_1125909_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/420767d4-810c-420a-a189-dc37b1096196_isrForCrosstalkSources_1125909_41.stderr parsl.dataflow.dflow INFO: Task 2508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e389ff415e2021926200a47bcb0fcc93 with result from task 2508 parsl.dataflow.dflow DEBUG: Task 2508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/e16d4912-8a40-469e-b305-4b0028915fed_isrForCrosstalkSources_1125906_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/e16d4912-8a40-469e-b305-4b0028915fed_isrForCrosstalkSources_1125906_24.stderr parsl.dataflow.dflow INFO: Task 2526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83e10268c32a1ee0a67d6c6fc96b7ef6 with result from task 2526 parsl.dataflow.dflow DEBUG: Task 2526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/144e6b6a-ca51-4d72-8fcc-7392dccbe9a5_isrForCrosstalkSources_1125929_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/144e6b6a-ca51-4d72-8fcc-7392dccbe9a5_isrForCrosstalkSources_1125929_2.stderr parsl.dataflow.dflow INFO: Task 2521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eea08e6c58fccd017a67d9b6fbd3c1be with result from task 2521 parsl.dataflow.dflow DEBUG: Task 2521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/65b6429e-5f23-43e9-9256-1625b7acb5d2_isrForCrosstalkSources_1125913_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/65b6429e-5f23-43e9-9256-1625b7acb5d2_isrForCrosstalkSources_1125913_3.stderr parsl.dataflow.dflow INFO: Task 2225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd9858e428714659f632fb89dde183ba with result from task 2225 parsl.dataflow.dflow DEBUG: Task 2225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/705fc3cd-3374-4623-bc33-0e4ea0d84184_isrForCrosstalkSources_1125939_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/705fc3cd-3374-4623-bc33-0e4ea0d84184_isrForCrosstalkSources_1125939_12.stderr parsl.dataflow.dflow INFO: Task 2499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8469caa7c58307b707e268de671ddb54 with result from task 2499 parsl.dataflow.dflow DEBUG: Task 2499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/bf0c9cda-b1ea-400e-8d0d-feea68729462_isrForCrosstalkSources_1125932_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/bf0c9cda-b1ea-400e-8d0d-feea68729462_isrForCrosstalkSources_1125932_49.stderr parsl.dataflow.dflow INFO: Task 2469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f4da9cec205234d1f5dff88c4d15adc with result from task 2469 parsl.dataflow.dflow DEBUG: Task 2469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/521a1fbb-28eb-4b79-b056-49669d8f78ef_isrForCrosstalkSources_1125902_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/521a1fbb-28eb-4b79-b056-49669d8f78ef_isrForCrosstalkSources_1125902_37.stderr parsl.dataflow.dflow INFO: Task 2502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa36f7db834819a3b396c7e660ad2a85 with result from task 2502 parsl.dataflow.dflow DEBUG: Task 2502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/d819c805-b474-4e56-8eb8-2c0c78e84223_isrForCrosstalkSources_1125935_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/d819c805-b474-4e56-8eb8-2c0c78e84223_isrForCrosstalkSources_1125935_50.stderr parsl.dataflow.dflow INFO: Task 2441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e7dde622a5c99ba15ad0e87b4eb7464 with result from task 2441 parsl.dataflow.dflow DEBUG: Task 2441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/521e0555-f72e-4e8e-9499-146358f7e009_isrForCrosstalkSources_1125913_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/521e0555-f72e-4e8e-9499-146358f7e009_isrForCrosstalkSources_1125913_2.stderr parsl.dataflow.dflow INFO: Task 2487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f0b7c85b8919361de3ae31a5efeeb9e with result from task 2487 parsl.dataflow.dflow DEBUG: Task 2487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/e4bde4d0-6752-4db6-a463-5f9737424703_isrForCrosstalkSources_1125922_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/e4bde4d0-6752-4db6-a463-5f9737424703_isrForCrosstalkSources_1125922_58.stderr parsl.dataflow.dflow INFO: Task 2255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fbc671f1ae567449d32780b0cc97a8a with result from task 2255 parsl.dataflow.dflow DEBUG: Task 2255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/21697b4e-df33-4fc5-90df-25f8f41290b4_isrForCrosstalkSources_1125890_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/21697b4e-df33-4fc5-90df-25f8f41290b4_isrForCrosstalkSources_1125890_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:53 proc_lsst.multi:146 [INFO] found job 16896420 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896420 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.dataflow.dflow INFO: Task 2415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c2d30efd7f64795a292354ad9c509b7 with result from task 2415 parsl.dataflow.dflow DEBUG: Task 2415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/65d6c609-ebb2-41a0-81ac-636563a38731_isrForCrosstalkSources_1125928_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/65d6c609-ebb2-41a0-81ac-636563a38731_isrForCrosstalkSources_1125928_53.stderr parsl.dataflow.dflow INFO: Task 2485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbda9f20f51a293d03c29bb16915f2a6 with result from task 2485 parsl.dataflow.dflow DEBUG: Task 2485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/76cf491d-a37b-48a5-afb6-74643f728fa4_isrForCrosstalkSources_1125899_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/76cf491d-a37b-48a5-afb6-74643f728fa4_isrForCrosstalkSources_1125899_46.stderr parsl.dataflow.dflow INFO: Task 2450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d8ffed1230dac63247aa93c096eb9a7 with result from task 2450 parsl.dataflow.dflow DEBUG: Task 2450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/0de08a5d-3153-440e-8b28-9483a14a9439_isrForCrosstalkSources_1125943_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/0de08a5d-3153-440e-8b28-9483a14a9439_isrForCrosstalkSources_1125943_3.stderr parsl.dataflow.dflow INFO: Task 2448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82fd566ce0418ae9558897eb0d08e4bc with result from task 2448 parsl.dataflow.dflow DEBUG: Task 2448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/228a358e-86b0-45b3-9f64-33221b68e704_isrForCrosstalkSources_1125902_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/228a358e-86b0-45b3-9f64-33221b68e704_isrForCrosstalkSources_1125902_3.stderr parsl.dataflow.dflow INFO: Task 2223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b40d10c801c13ab8fdc97eb214118e0 with result from task 2223 parsl.dataflow.dflow DEBUG: Task 2223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/291f7102-0635-4fc3-8bc6-a95a95555494_isrForCrosstalkSources_1125923_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/291f7102-0635-4fc3-8bc6-a95a95555494_isrForCrosstalkSources_1125923_62.stderr parsl.dataflow.dflow INFO: Task 2522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92ec593f7311d98d9b5d7113f5022563 with result from task 2522 parsl.dataflow.dflow DEBUG: Task 2522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/1ac32902-c64e-4df2-9a6e-a36e2e87c59e_isrForCrosstalkSources_1125933_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/1ac32902-c64e-4df2-9a6e-a36e2e87c59e_isrForCrosstalkSources_1125933_60.stderr parsl.dataflow.dflow INFO: Task 2355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06d371ad046be6dfafeeb6442babfa49 with result from task 2355 parsl.dataflow.dflow DEBUG: Task 2355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/cc617dbe-cc0c-4f93-bb0c-199bf437f6f4_isrForCrosstalkSources_1125941_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/cc617dbe-cc0c-4f93-bb0c-199bf437f6f4_isrForCrosstalkSources_1125941_59.stderr parsl.dataflow.dflow INFO: Task 2445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d20e495c36c2c17c883d4b0d6019944c with result from task 2445 parsl.dataflow.dflow DEBUG: Task 2445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/ea3fb236-4114-4239-bc57-52820533029d_isrForCrosstalkSources_1125934_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/ea3fb236-4114-4239-bc57-52820533029d_isrForCrosstalkSources_1125934_7.stderr parsl.dataflow.dflow INFO: Task 2168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28c3669abd53d530e95f009d04dc7736 with result from task 2168 parsl.dataflow.dflow DEBUG: Task 2168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/cb76ebca-2b16-4ed6-9ad2-602568af5b31_isrForCrosstalkSources_1125909_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/cb76ebca-2b16-4ed6-9ad2-602568af5b31_isrForCrosstalkSources_1125909_62.stderr parsl.dataflow.dflow INFO: Task 2528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 675777ced5ddae2e936f169e0d86eaa4 with result from task 2528 parsl.dataflow.dflow DEBUG: Task 2528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/46d135b5-5f14-44bd-b5a4-c8cfceea2f42_isrForCrosstalkSources_1125910_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/46d135b5-5f14-44bd-b5a4-c8cfceea2f42_isrForCrosstalkSources_1125910_7.stderr parsl.dataflow.dflow INFO: Task 2544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5505984ac1738260402beebe66fba1c4 with result from task 2544 parsl.dataflow.dflow DEBUG: Task 2544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/112f07d5-2dae-4007-96d4-fbec7942d516_isrForCrosstalkSources_1125943_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/112f07d5-2dae-4007-96d4-fbec7942d516_isrForCrosstalkSources_1125943_1.stderr parsl.dataflow.dflow INFO: Task 2492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 019e5834722d38489f41d8dcf6d18e78 with result from task 2492 parsl.dataflow.dflow DEBUG: Task 2492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/46b3790b-cfbf-449e-a525-86db50bd0ff3_isrForCrosstalkSources_1125894_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/46b3790b-cfbf-449e-a525-86db50bd0ff3_isrForCrosstalkSources_1125894_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:53 proc_lsst.multi:146 [INFO] found job 16896421 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896421 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.dataflow.dflow INFO: Task 2495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f41aac1bd1a9e2e79061dbd4f632b8eb with result from task 2495 parsl.dataflow.dflow DEBUG: Task 2495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/029ba425-fabc-4cfa-a3bf-289b9f9c0d2c_isrForCrosstalkSources_1125903_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/029ba425-fabc-4cfa-a3bf-289b9f9c0d2c_isrForCrosstalkSources_1125903_57.stderr parsl.dataflow.dflow INFO: Task 2159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d69403bf47e4057ceb05d68dddd7c8de with result from task 2159 parsl.dataflow.dflow DEBUG: Task 2159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/0e40d68a-b1e1-4f1c-8eb8-ad777fa9b90a_isrForCrosstalkSources_1125897_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/0e40d68a-b1e1-4f1c-8eb8-ad777fa9b90a_isrForCrosstalkSources_1125897_46.stderr parsl.dataflow.dflow INFO: Task 2489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1603f80dcf0e472e0ec06b75c884c34d with result from task 2489 parsl.dataflow.dflow DEBUG: Task 2489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/bee4bf02-711a-4a06-83e7-a4bfba29e491_isrForCrosstalkSources_1125929_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/bee4bf02-711a-4a06-83e7-a4bfba29e491_isrForCrosstalkSources_1125929_62.stderr parsl.dataflow.dflow INFO: Task 2209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bb4beeb4579d026c254a46fecb72904 with result from task 2209 parsl.dataflow.dflow DEBUG: Task 2209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/eb05f2c9-5f4e-489f-88ac-1157c0834d5b_isrForCrosstalkSources_1125936_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/eb05f2c9-5f4e-489f-88ac-1157c0834d5b_isrForCrosstalkSources_1125936_42.stderr parsl.dataflow.dflow INFO: Task 2516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f598c57f0ac7b8afd98a92da4b01645c with result from task 2516 parsl.dataflow.dflow DEBUG: Task 2516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/54700c93-2b0c-42a1-bf8d-875a58b00008_isrForCrosstalkSources_1125895_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/54700c93-2b0c-42a1-bf8d-875a58b00008_isrForCrosstalkSources_1125895_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:54 proc_lsst.multi:146 [INFO] found job 16896422 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896422 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.dataflow.dflow INFO: Task 2193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6898fb201110cb519c06b08562a6ee7 with result from task 2193 parsl.dataflow.dflow DEBUG: Task 2193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/7e2c6163-f8a9-42ec-bc87-facfcc9fbac6_isrForCrosstalkSources_1125943_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/7e2c6163-f8a9-42ec-bc87-facfcc9fbac6_isrForCrosstalkSources_1125943_59.stderr parsl.dataflow.dflow INFO: Task 2473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c00cffbb25e6860ba175253fab3b0855 with result from task 2473 parsl.dataflow.dflow DEBUG: Task 2473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/2f41b093-6f78-427e-8105-e7f10838c0d9_isrForCrosstalkSources_1125928_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/2f41b093-6f78-427e-8105-e7f10838c0d9_isrForCrosstalkSources_1125928_43.stderr parsl.dataflow.dflow INFO: Task 2406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b77a3f07679d7a9be4a4710b19bcb33 with result from task 2406 parsl.dataflow.dflow DEBUG: Task 2406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/f672bdae-c155-4713-92c6-a2a3b1fa3b60_isrForCrosstalkSources_1125901_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/f672bdae-c155-4713-92c6-a2a3b1fa3b60_isrForCrosstalkSources_1125901_62.stderr parsl.dataflow.dflow INFO: Task 2545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0c35d43e5c23edd8aea6d9761f33474 with result from task 2545 parsl.dataflow.dflow DEBUG: Task 2545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/f786fdad-9902-40bb-befa-24a57e596f3f_isrForCrosstalkSources_1125920_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/f786fdad-9902-40bb-befa-24a57e596f3f_isrForCrosstalkSources_1125920_45.stderr parsl.dataflow.dflow INFO: Task 2184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c9988819b681dee9b5ee07eb2e1ca34 with result from task 2184 parsl.dataflow.dflow DEBUG: Task 2184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/b8235e23-faf7-4d05-bb01-85cb683d69e7_isrForCrosstalkSources_1125918_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/b8235e23-faf7-4d05-bb01-85cb683d69e7_isrForCrosstalkSources_1125918_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:54 proc_lsst.multi:146 [INFO] found job 16896423 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896423 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896419,16896420,16896421,16896422,16896423' parsl.dataflow.dflow INFO: Task 2307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58be8aa93a85aacf97d9d05c7394df0f with result from task 2307 parsl.dataflow.dflow DEBUG: Task 2307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/02ddcd9f-32d7-41c5-8c07-104af8d3c53c_isrForCrosstalkSources_1125945_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/02ddcd9f-32d7-41c5-8c07-104af8d3c53c_isrForCrosstalkSources_1125945_62.stderr parsl.dataflow.dflow INFO: Task 2501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf01aac9bab26c676fc8e12f6d29c26d with result from task 2501 parsl.dataflow.dflow DEBUG: Task 2501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/29c9d42e-39eb-48a5-b94a-d0fa2688cfd4_isrForCrosstalkSources_1125913_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/29c9d42e-39eb-48a5-b94a-d0fa2688cfd4_isrForCrosstalkSources_1125913_37.stderr parsl.dataflow.dflow INFO: Task 2455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 117a07c7eb894ceecd8d6b9aef6dd554 with result from task 2455 parsl.dataflow.dflow DEBUG: Task 2455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6ed79345-6a57-4a6c-887a-b33b5811213a_isrForCrosstalkSources_1125927_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6ed79345-6a57-4a6c-887a-b33b5811213a_isrForCrosstalkSources_1125927_61.stderr parsl.dataflow.dflow INFO: Task 2509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1e139976b1c655a50b867262388946b with result from task 2509 parsl.dataflow.dflow DEBUG: Task 2509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ba716670-ba5c-4132-bc34-7ab88f434b57_isrForCrosstalkSources_1125947_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/ba716670-ba5c-4132-bc34-7ab88f434b57_isrForCrosstalkSources_1125947_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896419 R 16896420 R 16896421 R 16896422 R 16896423 R parsl.providers.slurm.slurm DEBUG: Updating job 16896419 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896420 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896421 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896422 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896423 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:54 proc_lsst.multi:146 [INFO] found job 16896424 in provider escience_a40 proc_lsst.multi INFO: found job 16896424 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 076d6abe23421a32900447b3635fcf97 with result from task 2382 parsl.dataflow.dflow DEBUG: Task 2382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/d20c75c0-9795-40be-aa58-bd15e8b32de5_isrForCrosstalkSources_1125910_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/d20c75c0-9795-40be-aa58-bd15e8b32de5_isrForCrosstalkSources_1125910_57.stderr parsl.dataflow.dflow INFO: Task 2507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31c2b644b5b0e8aaea98decae0dbe279 with result from task 2507 parsl.dataflow.dflow DEBUG: Task 2507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/7a95c313-c882-4eae-b6ef-eb09cf6f1f81_isrForCrosstalkSources_1125921_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/7a95c313-c882-4eae-b6ef-eb09cf6f1f81_isrForCrosstalkSources_1125921_38.stderr parsl.dataflow.dflow INFO: Task 2403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 464190146213162bfe66b9a7314d392e with result from task 2403 parsl.dataflow.dflow DEBUG: Task 2403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/5cdacb36-f94e-4760-bc7d-748a686c591d_isrForCrosstalkSources_1125888_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/5cdacb36-f94e-4760-bc7d-748a686c591d_isrForCrosstalkSources_1125888_2.stderr parsl.dataflow.dflow INFO: Task 2538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9540f1de5b49ade49665f79f00f5d20c with result from task 2538 parsl.dataflow.dflow DEBUG: Task 2538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/dca9be6a-76be-4be3-bc5a-ae2eea0295fa_isrForCrosstalkSources_1125896_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/dca9be6a-76be-4be3-bc5a-ae2eea0295fa_isrForCrosstalkSources_1125896_50.stderr parsl.dataflow.dflow INFO: Task 2532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66d2ac731d8ec038adcf13f424a9da28 with result from task 2532 parsl.dataflow.dflow DEBUG: Task 2532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/41abb216-fc87-47b4-be0c-f35025a2378f_isrForCrosstalkSources_1125912_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/41abb216-fc87-47b4-be0c-f35025a2378f_isrForCrosstalkSources_1125912_29.stderr parsl.dataflow.dflow INFO: Task 2379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e91cbfd060f9badaf71bad14ee6c9df0 with result from task 2379 parsl.dataflow.dflow DEBUG: Task 2379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/a3ad8fc0-cd5b-4128-b83d-ed8365f94a9e_isrForCrosstalkSources_1125942_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/a3ad8fc0-cd5b-4128-b83d-ed8365f94a9e_isrForCrosstalkSources_1125942_29.stderr parsl.dataflow.dflow INFO: Task 2511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4467e5b897911025105c2ec8a0be1240 with result from task 2511 parsl.dataflow.dflow DEBUG: Task 2511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/3ec15a3d-5016-4275-ac9c-ff5707232165_isrForCrosstalkSources_1125926_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/3ec15a3d-5016-4275-ac9c-ff5707232165_isrForCrosstalkSources_1125926_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:54 proc_lsst.multi:146 [INFO] found job 16896425 in provider escience_a40 proc_lsst.multi INFO: found job 16896425 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8eeed609a8da6095ac82c82cb46b7cda with result from task 2398 parsl.dataflow.dflow DEBUG: Task 2398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/f0dcbfa9-0e8b-4936-a6bf-fb515d2d8a37_isrForCrosstalkSources_1125937_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/f0dcbfa9-0e8b-4936-a6bf-fb515d2d8a37_isrForCrosstalkSources_1125937_50.stderr parsl.dataflow.dflow INFO: Task 2530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bf449f9d1c02c5a1ad589d619283b11 with result from task 2530 parsl.dataflow.dflow DEBUG: Task 2530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/8088a4ce-070b-4ab1-aded-146db0f5df96_isrForCrosstalkSources_1125896_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/8088a4ce-070b-4ab1-aded-146db0f5df96_isrForCrosstalkSources_1125896_55.stderr parsl.dataflow.dflow INFO: Task 2262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9c96577675a3eae008123d49c0b07ad with result from task 2262 parsl.dataflow.dflow DEBUG: Task 2262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/b50eeb99-37f1-4718-b4d9-6229db89fa19_isrForCrosstalkSources_1125895_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/b50eeb99-37f1-4718-b4d9-6229db89fa19_isrForCrosstalkSources_1125895_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:54 proc_lsst.multi:146 [INFO] found job 16896426 in provider escience_a40 proc_lsst.multi INFO: found job 16896426 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25ec6961034f273cbe447f6e0d7dbcf8 with result from task 2394 parsl.dataflow.dflow DEBUG: Task 2394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/621e973f-eb18-4620-bde4-b03601aa6306_isrForCrosstalkSources_1125929_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/621e973f-eb18-4620-bde4-b03601aa6306_isrForCrosstalkSources_1125929_59.stderr parsl.dataflow.dflow INFO: Task 2555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 286aeef9767c3c1e2a26b2afc2e01230 with result from task 2555 parsl.dataflow.dflow DEBUG: Task 2555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/0167ecd7-e74b-4838-a6c3-8108181e53af_isrForCrosstalkSources_1125929_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/0167ecd7-e74b-4838-a6c3-8108181e53af_isrForCrosstalkSources_1125929_27.stderr parsl.dataflow.dflow INFO: Task 2563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e91e7121e90728b8ec5c254a611c442 with result from task 2563 parsl.dataflow.dflow DEBUG: Task 2563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/6f35a36c-32ee-4568-8a0c-36fe9cb4ea2a_isrForCrosstalkSources_1125945_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/6f35a36c-32ee-4568-8a0c-36fe9cb4ea2a_isrForCrosstalkSources_1125945_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:55 proc_lsst.multi:146 [INFO] found job 16896427 in provider escience_a40 proc_lsst.multi INFO: found job 16896427 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f6d3530e7b7ffba320310f201327d02 with result from task 2592 parsl.dataflow.dflow DEBUG: Task 2592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/b078ae62-3f94-4dc2-9160-93c6ac3bc782_isrForCrosstalkSources_1125920_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/b078ae62-3f94-4dc2-9160-93c6ac3bc782_isrForCrosstalkSources_1125920_16.stderr parsl.dataflow.dflow INFO: Task 2550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e243b372a7bdcb3dd9e9200bc1640549 with result from task 2550 parsl.dataflow.dflow DEBUG: Task 2550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9ed5a1e6-1ad6-43d6-9a49-08122e1c568b_isrForCrosstalkSources_1125934_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9ed5a1e6-1ad6-43d6-9a49-08122e1c568b_isrForCrosstalkSources_1125934_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:55 proc_lsst.multi:146 [INFO] found job 16896428 in provider escience_a40 proc_lsst.multi INFO: found job 16896428 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3de405aaf6a097059771b1152186aa6 with result from task 2233 parsl.dataflow.dflow DEBUG: Task 2233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/1859a85f-753f-40fe-be5b-d0d6ee7caaca_isrForCrosstalkSources_1125908_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/1859a85f-753f-40fe-be5b-d0d6ee7caaca_isrForCrosstalkSources_1125908_50.stderr parsl.dataflow.dflow INFO: Task 2466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff7afdf188f95c021e317de76948ade1 with result from task 2466 parsl.dataflow.dflow DEBUG: Task 2466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/94cc673a-da17-45a9-b7fc-088bd2a4a75f_isrForCrosstalkSources_1125927_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/94cc673a-da17-45a9-b7fc-088bd2a4a75f_isrForCrosstalkSources_1125927_25.stderr parsl.dataflow.dflow INFO: Task 2426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82a6258c1d5059fb7668e3386dbfe571 with result from task 2426 parsl.dataflow.dflow DEBUG: Task 2426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d043eb23-5dd3-4348-8420-85cb7070b9f4_isrForCrosstalkSources_1125919_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d043eb23-5dd3-4348-8420-85cb7070b9f4_isrForCrosstalkSources_1125919_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:55 proc_lsst.multi:146 [INFO] found job 16896429 in provider escience_a40 proc_lsst.multi INFO: found job 16896429 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c46f667aa1e9f85e6369f9f568a27450 with result from task 2552 parsl.dataflow.dflow DEBUG: Task 2552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/1bddabb1-0ff1-4ac0-9d83-f20742c5df63_isrForCrosstalkSources_1125930_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/1bddabb1-0ff1-4ac0-9d83-f20742c5df63_isrForCrosstalkSources_1125930_33.stderr parsl.dataflow.dflow INFO: Task 2572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a89ceb3ce851a3af6544c797aeba03e with result from task 2572 parsl.dataflow.dflow DEBUG: Task 2572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/d8335c3b-7c76-4974-aaa3-7a1ee05569c6_isrForCrosstalkSources_1125930_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/d8335c3b-7c76-4974-aaa3-7a1ee05569c6_isrForCrosstalkSources_1125930_10.stderr parsl.dataflow.dflow INFO: Task 2553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ba73aaad8b8f22d914f7503188c3d26 with result from task 2553 parsl.dataflow.dflow DEBUG: Task 2553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/b9c24582-d89d-4fad-9557-f19cc28857d5_isrForCrosstalkSources_1125928_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/b9c24582-d89d-4fad-9557-f19cc28857d5_isrForCrosstalkSources_1125928_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:55 proc_lsst.multi:146 [INFO] found job 16896430 in provider escience_a40 proc_lsst.multi INFO: found job 16896430 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac0b5cc351eaee6adeb5cb2903d1f45e with result from task 2546 parsl.dataflow.dflow DEBUG: Task 2546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/984dd845-afff-4383-8e6a-58d8135a991a_isrForCrosstalkSources_1125893_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/984dd845-afff-4383-8e6a-58d8135a991a_isrForCrosstalkSources_1125893_56.stderr parsl.dataflow.dflow INFO: Task 2384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d400f30a3143089a0899f6880528fb02 with result from task 2384 parsl.dataflow.dflow DEBUG: Task 2384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/c6af7e6a-2885-404d-95fc-04fd73037174_isrForCrosstalkSources_1125924_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/c6af7e6a-2885-404d-95fc-04fd73037174_isrForCrosstalkSources_1125924_56.stderr parsl.dataflow.dflow INFO: Task 2481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63ae9fec63d3735e959cafa70702fa6d with result from task 2481 parsl.dataflow.dflow DEBUG: Task 2481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/115b6b41-d050-4fd1-934a-891cfbe0d65a_isrForCrosstalkSources_1125922_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/115b6b41-d050-4fd1-934a-891cfbe0d65a_isrForCrosstalkSources_1125922_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:55 proc_lsst.multi:146 [INFO] found job 16896431 in provider escience_a40 proc_lsst.multi INFO: found job 16896431 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d7329278a1ee72563ee55d7259bbae2 with result from task 2548 parsl.dataflow.dflow DEBUG: Task 2548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/191bad66-7490-49c1-8772-8cd48cf56750_isrForCrosstalkSources_1125892_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/191bad66-7490-49c1-8772-8cd48cf56750_isrForCrosstalkSources_1125892_1.stderr parsl.dataflow.dflow INFO: Task 2590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46026d6aee7c5c1a93f268556c9abbe0 with result from task 2590 parsl.dataflow.dflow DEBUG: Task 2590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/df9b56fb-1d28-4cbc-ab0d-75b3dad48fa2_isrForCrosstalkSources_1125944_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/df9b56fb-1d28-4cbc-ab0d-75b3dad48fa2_isrForCrosstalkSources_1125944_8.stderr parsl.dataflow.dflow INFO: Task 2542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2632c1e917e30ca23d138b299695713 with result from task 2542 parsl.dataflow.dflow DEBUG: Task 2542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/96171b26-176a-463c-a0e6-eb1e4e690349_isrForCrosstalkSources_1125908_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/96171b26-176a-463c-a0e6-eb1e4e690349_isrForCrosstalkSources_1125908_14.stderr parsl.dataflow.dflow INFO: Task 2541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d138aeeffdb50cf5a9b456adb8590b61 with result from task 2541 parsl.dataflow.dflow DEBUG: Task 2541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/15076fc2-558e-44ec-bd64-e787fa69ede8_isrForCrosstalkSources_1125906_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/15076fc2-558e-44ec-bd64-e787fa69ede8_isrForCrosstalkSources_1125906_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:56 proc_lsst.multi:146 [INFO] found job 16896432 in provider escience_a40 proc_lsst.multi INFO: found job 16896432 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0c700c2f2b8fbdb7a4861e1e718c57f with result from task 2531 parsl.dataflow.dflow DEBUG: Task 2531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/2a29c9ed-86ec-4e12-ab57-389869ba24a5_isrForCrosstalkSources_1125918_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/2a29c9ed-86ec-4e12-ab57-389869ba24a5_isrForCrosstalkSources_1125918_56.stderr parsl.dataflow.dflow INFO: Task 2543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25b415081aa6beca52593590693cd47d with result from task 2543 parsl.dataflow.dflow DEBUG: Task 2543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/db2638f6-1c93-4415-b8dd-e67f14ccf471_isrForCrosstalkSources_1125923_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/db2638f6-1c93-4415-b8dd-e67f14ccf471_isrForCrosstalkSources_1125923_13.stderr parsl.dataflow.dflow INFO: Task 2595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1fc2a34249253d0909865c0ed2d47dc with result from task 2595 parsl.dataflow.dflow DEBUG: Task 2595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/33216f62-04c5-4d41-88a9-b6558bcc4bac_isrForCrosstalkSources_1125931_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/33216f62-04c5-4d41-88a9-b6558bcc4bac_isrForCrosstalkSources_1125931_16.stderr parsl.dataflow.dflow INFO: Task 2554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 816bcb149a128027679bd1a23a6efbb9 with result from task 2554 parsl.dataflow.dflow DEBUG: Task 2554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/efc06221-15ed-45a6-a7f9-ebbec51867ff_isrForCrosstalkSources_1125890_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/efc06221-15ed-45a6-a7f9-ebbec51867ff_isrForCrosstalkSources_1125890_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:56 proc_lsst.multi:146 [INFO] found job 16896433 in provider escience_a40 proc_lsst.multi INFO: found job 16896433 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53d4d2a84d9ef64ae6331e612fc65509 with result from task 2570 parsl.dataflow.dflow DEBUG: Task 2570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/1043d64a-f2f6-42cd-9751-7b7e5ceeff2d_isrForCrosstalkSources_1125938_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/1043d64a-f2f6-42cd-9751-7b7e5ceeff2d_isrForCrosstalkSources_1125938_61.stderr parsl.dataflow.dflow INFO: Task 2470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fce3a5807dce6991aeee07b4bbe446fc with result from task 2470 parsl.dataflow.dflow DEBUG: Task 2470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/43b4921a-3bd3-47a2-9dbb-91d54e1a97b0_isrForCrosstalkSources_1125900_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/43b4921a-3bd3-47a2-9dbb-91d54e1a97b0_isrForCrosstalkSources_1125900_17.stderr parsl.dataflow.dflow INFO: Task 2520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5687f67a1595b80578a478a3f6bc693 with result from task 2520 parsl.dataflow.dflow DEBUG: Task 2520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ad442d0c-b651-4861-a387-f2d7b7d531fc_isrForCrosstalkSources_1125939_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ad442d0c-b651-4861-a387-f2d7b7d531fc_isrForCrosstalkSources_1125939_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:56 proc_lsst.multi:146 [INFO] found job 16896434 in provider escience_a40 proc_lsst.multi INFO: found job 16896434 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b24b53a9c285e01c6d7b14f73c7c6c0 with result from task 2482 parsl.dataflow.dflow DEBUG: Task 2482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/3f2f922b-a7eb-4f00-8a74-5d8e2833fcd7_isrForCrosstalkSources_1125938_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/3f2f922b-a7eb-4f00-8a74-5d8e2833fcd7_isrForCrosstalkSources_1125938_45.stderr parsl.dataflow.dflow INFO: Task 2540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 897ec9723028779a7f67ffa16b30f1ff with result from task 2540 parsl.dataflow.dflow DEBUG: Task 2540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/fe969b72-4e65-4613-a851-1830b2b3a476_isrForCrosstalkSources_1125932_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/fe969b72-4e65-4613-a851-1830b2b3a476_isrForCrosstalkSources_1125932_23.stderr parsl.dataflow.dflow INFO: Task 2549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f28cee63fe14f3cbaa62ac058b05bc80 with result from task 2549 parsl.dataflow.dflow DEBUG: Task 2549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/926771e2-b228-40c8-af7a-1a853f5caefc_isrForCrosstalkSources_1125920_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/926771e2-b228-40c8-af7a-1a853f5caefc_isrForCrosstalkSources_1125920_36.stderr parsl.dataflow.dflow INFO: Task 2536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96f66a70996e61d2757eed960088f1b1 with result from task 2536 parsl.dataflow.dflow DEBUG: Task 2536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/7cdf782b-52e6-410f-bf4b-91717696acb0_isrForCrosstalkSources_1125897_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/7cdf782b-52e6-410f-bf4b-91717696acb0_isrForCrosstalkSources_1125897_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:57 proc_lsst.multi:146 [INFO] found job 16896435 in provider escience_a40 proc_lsst.multi INFO: found job 16896435 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2417cdd03bd41b1a598fc1788a7b651f with result from task 2547 parsl.dataflow.dflow DEBUG: Task 2547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/d5d6258d-1c90-4df7-9e9f-85c9f49ae570_isrForCrosstalkSources_1125898_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/d5d6258d-1c90-4df7-9e9f-85c9f49ae570_isrForCrosstalkSources_1125898_5.stderr parsl.dataflow.dflow INFO: Task 2535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24c27b74dedec2943ce84cd1ce04299c with result from task 2535 parsl.dataflow.dflow DEBUG: Task 2535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/328c142f-74b0-4df6-a759-79a33283258a_isrForCrosstalkSources_1125908_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/328c142f-74b0-4df6-a759-79a33283258a_isrForCrosstalkSources_1125908_60.stderr parsl.dataflow.dflow INFO: Task 2330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d9f94ab6f15939269c4eeb3a2c5681f with result from task 2330 parsl.dataflow.dflow DEBUG: Task 2330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/6324ee1e-8fd4-4bd5-8d89-b9ecbd39f4d3_isrForCrosstalkSources_1125892_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/6324ee1e-8fd4-4bd5-8d89-b9ecbd39f4d3_isrForCrosstalkSources_1125892_25.stderr parsl.dataflow.dflow INFO: Task 2605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d257ae2820d8f0bd1f9d3bb2466f0be1 with result from task 2605 parsl.dataflow.dflow DEBUG: Task 2605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/8d6f12d9-a2d0-42e1-8ed9-3adc1a073eae_isrForCrosstalkSources_1125920_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/8d6f12d9-a2d0-42e1-8ed9-3adc1a073eae_isrForCrosstalkSources_1125920_22.stderr parsl.dataflow.dflow INFO: Task 2412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40ad774e174211f9ca158550ea23a3cb with result from task 2412 parsl.dataflow.dflow DEBUG: Task 2412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/50da7ea6-2b59-4f45-b92a-c5d378d19a07_isrForCrosstalkSources_1125889_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/50da7ea6-2b59-4f45-b92a-c5d378d19a07_isrForCrosstalkSources_1125889_44.stderr parsl.dataflow.dflow INFO: Task 2476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 906e7110e4a26bac8ac32576eae91c88 with result from task 2476 parsl.dataflow.dflow DEBUG: Task 2476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/9bd9fb08-e454-4c69-816a-6b9332fb716a_isrForCrosstalkSources_1125919_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/9bd9fb08-e454-4c69-816a-6b9332fb716a_isrForCrosstalkSources_1125919_43.stderr parsl.dataflow.dflow INFO: Task 2610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9df27037c7c4a2ba605341c47e2cf69f with result from task 2610 parsl.dataflow.dflow DEBUG: Task 2610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/7e2ba598-e218-44f8-a07c-ab1d251b2e07_isrForCrosstalkSources_1125891_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/7e2ba598-e218-44f8-a07c-ab1d251b2e07_isrForCrosstalkSources_1125891_3.stderr parsl.dataflow.dflow INFO: Task 2468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc631a221cbf6efe72863094f0692cef with result from task 2468 parsl.dataflow.dflow DEBUG: Task 2468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/fb197848-6368-4099-86ae-4be5c3486fc9_isrForCrosstalkSources_1125935_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/fb197848-6368-4099-86ae-4be5c3486fc9_isrForCrosstalkSources_1125935_47.stderr parsl.dataflow.dflow INFO: Task 2261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3a4eb290e659e9b70c75dd22164536b with result from task 2261 parsl.dataflow.dflow DEBUG: Task 2261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/106f4519-82ff-4e32-ab7d-72ed5149e171_isrForCrosstalkSources_1125940_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/106f4519-82ff-4e32-ab7d-72ed5149e171_isrForCrosstalkSources_1125940_35.stderr parsl.dataflow.dflow INFO: Task 2602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f2f58b9f9d33b989654d2e74f3eacc4 with result from task 2602 parsl.dataflow.dflow DEBUG: Task 2602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/fd5aeb53-413f-4af7-9922-9c63f6d67ddc_isrForCrosstalkSources_1125892_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/fd5aeb53-413f-4af7-9922-9c63f6d67ddc_isrForCrosstalkSources_1125892_61.stderr parsl.dataflow.dflow INFO: Task 2517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c6bb005dd6c7643a3000bc9f2a4de09 with result from task 2517 parsl.dataflow.dflow DEBUG: Task 2517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/22a20b49-55fe-4788-935c-b9e1bc7b18e1_isrForCrosstalkSources_1125941_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/22a20b49-55fe-4788-935c-b9e1bc7b18e1_isrForCrosstalkSources_1125941_23.stderr parsl.dataflow.dflow INFO: Task 2539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 819d8da36ed81c270bbfb19fa0e0e3c0 with result from task 2539 parsl.dataflow.dflow DEBUG: Task 2539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/46adedc5-9faf-4646-b62e-b4bb42bce309_isrForCrosstalkSources_1125943_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/46adedc5-9faf-4646-b62e-b4bb42bce309_isrForCrosstalkSources_1125943_50.stderr parsl.dataflow.dflow INFO: Task 2491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1732c9a91de7678140cfd02f2699b43f with result from task 2491 parsl.dataflow.dflow DEBUG: Task 2491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/950ab69e-a603-412e-bcbf-9c9d669a2b58_isrForCrosstalkSources_1125907_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/950ab69e-a603-412e-bcbf-9c9d669a2b58_isrForCrosstalkSources_1125907_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:57 proc_lsst.multi:146 [INFO] found job 16896437 in provider escience_a40 proc_lsst.multi INFO: found job 16896437 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ef5670ab91310415a1b313e4c447e24 with result from task 2612 parsl.dataflow.dflow DEBUG: Task 2612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/784673cb-bb3f-4159-a0e5-c182390aeb05_isrForCrosstalkSources_1125893_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/784673cb-bb3f-4159-a0e5-c182390aeb05_isrForCrosstalkSources_1125893_47.stderr parsl.dataflow.dflow INFO: Task 2564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45fcd95362553490b2ab37b8e945e353 with result from task 2564 parsl.dataflow.dflow DEBUG: Task 2564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/13c38b60-e34d-4a9e-b659-a6a058844093_isrForCrosstalkSources_1125944_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/13c38b60-e34d-4a9e-b659-a6a058844093_isrForCrosstalkSources_1125944_55.stderr parsl.dataflow.dflow INFO: Task 2556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84c8bd06f74a276767d19589b5222642 with result from task 2556 parsl.dataflow.dflow DEBUG: Task 2556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/3b34953c-8c50-4ac9-9783-0c7d64b8ff1e_isrForCrosstalkSources_1125933_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/3b34953c-8c50-4ac9-9783-0c7d64b8ff1e_isrForCrosstalkSources_1125933_50.stderr parsl.dataflow.dflow INFO: Task 2557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf64b0df06fcb7a63e9f612ade6a913f with result from task 2557 parsl.dataflow.dflow DEBUG: Task 2557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/8befcf5a-91f2-41fa-a504-6fca966fe58a_isrForCrosstalkSources_1125901_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/8befcf5a-91f2-41fa-a504-6fca966fe58a_isrForCrosstalkSources_1125901_11.stderr parsl.dataflow.dflow INFO: Task 2551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c29f6074cafe7b89c88b44824334e057 with result from task 2551 parsl.dataflow.dflow DEBUG: Task 2551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/50e31278-e651-4427-a6e2-29f051121416_isrForCrosstalkSources_1125934_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/50e31278-e651-4427-a6e2-29f051121416_isrForCrosstalkSources_1125934_61.stderr parsl.dataflow.dflow INFO: Task 2597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bd23afb5d34d6d2c2be8fcaf2682723 with result from task 2597 parsl.dataflow.dflow DEBUG: Task 2597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c99b7296-4f18-435b-87ea-0ed8c00b1abc_isrForCrosstalkSources_1125893_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c99b7296-4f18-435b-87ea-0ed8c00b1abc_isrForCrosstalkSources_1125893_41.stderr parsl.dataflow.dflow INFO: Task 2566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c6cf6b1e746f1bd28525aaa3dfbdb75 with result from task 2566 parsl.dataflow.dflow DEBUG: Task 2566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/24018173-820a-4691-bea7-747542a20100_isrForCrosstalkSources_1125918_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/24018173-820a-4691-bea7-747542a20100_isrForCrosstalkSources_1125918_3.stderr parsl.dataflow.dflow INFO: Task 2626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf7d9406c428f547fc50e3856256ad60 with result from task 2626 parsl.dataflow.dflow DEBUG: Task 2626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/090f92aa-bce2-4fea-893e-3982e9dd36c5_isrForCrosstalkSources_1125935_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/090f92aa-bce2-4fea-893e-3982e9dd36c5_isrForCrosstalkSources_1125935_23.stderr parsl.dataflow.dflow INFO: Task 2627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8e4cc985b528ab8b02792691a5d0fb6 with result from task 2627 parsl.dataflow.dflow DEBUG: Task 2627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/bac2a3a2-4d9c-42ee-b344-f9cf33313e77_isrForCrosstalkSources_1125902_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/bac2a3a2-4d9c-42ee-b344-f9cf33313e77_isrForCrosstalkSources_1125902_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:58 proc_lsst.multi:146 [INFO] found job 16896438 in provider escience_a40 proc_lsst.multi INFO: found job 16896438 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a2c108ac9b01d5a11ce8bcd91819f9a with result from task 2577 parsl.dataflow.dflow DEBUG: Task 2577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/6081de18-f174-4e94-9584-2ffffd77e0ef_isrForCrosstalkSources_1125935_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/6081de18-f174-4e94-9584-2ffffd77e0ef_isrForCrosstalkSources_1125935_21.stderr parsl.dataflow.dflow INFO: Task 2567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c373ceb8160814bd9e3d1b18a8743bf1 with result from task 2567 parsl.dataflow.dflow DEBUG: Task 2567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/bd0d4298-4fca-4232-b08b-ada6ca1fc3d2_isrForCrosstalkSources_1125916_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/bd0d4298-4fca-4232-b08b-ada6ca1fc3d2_isrForCrosstalkSources_1125916_38.stderr parsl.dataflow.dflow INFO: Task 2593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c50cc850609b2aea737966863165d86a with result from task 2593 parsl.dataflow.dflow DEBUG: Task 2593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/a373be8d-5b09-421f-ae6d-cc9761ec849c_isrForCrosstalkSources_1125889_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/a373be8d-5b09-421f-ae6d-cc9761ec849c_isrForCrosstalkSources_1125889_35.stderr parsl.dataflow.dflow INFO: Task 2376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a102af86395677f9e459a33005251bb with result from task 2376 parsl.dataflow.dflow DEBUG: Task 2376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/9d54c593-f9c4-4fa2-9ff8-f17c6a1f96c8_isrForCrosstalkSources_1125919_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/9d54c593-f9c4-4fa2-9ff8-f17c6a1f96c8_isrForCrosstalkSources_1125919_22.stderr parsl.dataflow.dflow INFO: Task 2629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a344577211436bf96232875b4dfcb70 with result from task 2629 parsl.dataflow.dflow DEBUG: Task 2629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/27b9a12c-8cd3-47b8-bcb3-82c6ebe4b84e_isrForCrosstalkSources_1125899_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/27b9a12c-8cd3-47b8-bcb3-82c6ebe4b84e_isrForCrosstalkSources_1125899_53.stderr parsl.dataflow.dflow INFO: Task 2604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 038179bc21ecf16ac4e4689410ed480d with result from task 2604 parsl.dataflow.dflow DEBUG: Task 2604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/582ddf64-4d91-494d-b35e-5228492f2b5a_isrForCrosstalkSources_1125891_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/582ddf64-4d91-494d-b35e-5228492f2b5a_isrForCrosstalkSources_1125891_4.stderr parsl.dataflow.dflow INFO: Task 2582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e36a11c247dd041b9e980aff824afb4a with result from task 2582 parsl.dataflow.dflow DEBUG: Task 2582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a306c2e3-5aeb-47b5-a5a2-6c0f4f49f7de_isrForCrosstalkSources_1125900_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a306c2e3-5aeb-47b5-a5a2-6c0f4f49f7de_isrForCrosstalkSources_1125900_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:58 proc_lsst.multi:146 [INFO] found job 16896439 in provider escience_a40 proc_lsst.multi INFO: found job 16896439 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aedf48803bdb9dacf56fd249930ba1b0 with result from task 2586 parsl.dataflow.dflow DEBUG: Task 2586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/e82dde5e-9326-44ec-896a-9ced353bae1c_isrForCrosstalkSources_1125941_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/e82dde5e-9326-44ec-896a-9ced353bae1c_isrForCrosstalkSources_1125941_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:59 proc_lsst.multi:146 [INFO] found job 16896440 in provider escience_a40 proc_lsst.multi INFO: found job 16896440 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ee7843cb6613bddc8d9e1f0cf288114 with result from task 2591 parsl.dataflow.dflow DEBUG: Task 2591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/46edbd82-7ea3-493e-863f-6e825a844bb9_isrForCrosstalkSources_1125940_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/46edbd82-7ea3-493e-863f-6e825a844bb9_isrForCrosstalkSources_1125940_21.stderr parsl.dataflow.dflow INFO: Task 2574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b3d67d7625a8686154bca08ca083093 with result from task 2574 parsl.dataflow.dflow DEBUG: Task 2574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/48bb9258-ec16-4fb6-a804-8f070fba57dd_isrForCrosstalkSources_1125942_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/48bb9258-ec16-4fb6-a804-8f070fba57dd_isrForCrosstalkSources_1125942_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:59 proc_lsst.multi:146 [INFO] found job 16896441 in provider escience_a40 proc_lsst.multi INFO: found job 16896441 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c4fb015aa59c416624a7c58a69b5430 with result from task 2558 parsl.dataflow.dflow DEBUG: Task 2558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/9b88cf1c-efe8-41e3-9961-e2ce74624642_isrForCrosstalkSources_1125912_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/9b88cf1c-efe8-41e3-9961-e2ce74624642_isrForCrosstalkSources_1125912_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:59 proc_lsst.multi:146 [INFO] found job 16896442 in provider escience_a40 proc_lsst.multi INFO: found job 16896442 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 047a64a71ce0dd7889d0d67b40f96f34 with result from task 2573 parsl.dataflow.dflow DEBUG: Task 2573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/85c10834-ff9c-4740-a0aa-51e62eeedb7d_isrForCrosstalkSources_1125896_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/85c10834-ff9c-4740-a0aa-51e62eeedb7d_isrForCrosstalkSources_1125896_15.stderr parsl.dataflow.dflow INFO: Task 2654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ac301291ded531f69c6f6d861f27a3e with result from task 2654 parsl.dataflow.dflow DEBUG: Task 2654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f01363e1-2d93-4f7a-9f79-fe9ac81f5a55_isrForCrosstalkSources_1125930_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/f01363e1-2d93-4f7a-9f79-fe9ac81f5a55_isrForCrosstalkSources_1125930_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:59 proc_lsst.multi:146 [INFO] found job 16896443 in provider escience_a40 proc_lsst.multi INFO: found job 16896443 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:13:59 proc_lsst.multi:146 [INFO] found job 16896444 in provider escience_a40 proc_lsst.multi INFO: found job 16896444 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896424,16896425,16896426,16896427,16896428,16896429,16896430,16896431,16896432,16896433,16896434,16896435,16896437,16896438,16896439,16896440,16896441,16896442,16896443,16896444' parsl.dataflow.dflow INFO: Task 2584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9833f39f14d9cfa940ebb0ec4c313da9 with result from task 2584 parsl.dataflow.dflow DEBUG: Task 2584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/c3edee0b-d049-4b1e-acbd-49dfb1a51c91_isrForCrosstalkSources_1125920_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/c3edee0b-d049-4b1e-acbd-49dfb1a51c91_isrForCrosstalkSources_1125920_41.stderr parsl.dataflow.dflow INFO: Task 2619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da998a37ba6974d41dd80d1ed236addf with result from task 2619 parsl.dataflow.dflow DEBUG: Task 2619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/8861d821-f52d-4e96-b5d6-8acf2257c125_isrForCrosstalkSources_1125914_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/8861d821-f52d-4e96-b5d6-8acf2257c125_isrForCrosstalkSources_1125914_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896440 R 16896441 R 16896442 R 16896443 R 16896444 R 16896430 R 16896431 R 16896432 R 16896433 R 16896434 R 16896435 R 16896437 R 16896438 R 16896439 R 16896424 R 16896425 R 16896426 R 16896427 R 16896428 R 16896429 R parsl.providers.slurm.slurm DEBUG: Updating job 16896440 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896441 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896442 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896443 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896444 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896430 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896431 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896432 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896433 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896434 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896435 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896437 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896438 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896439 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896424 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896425 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896426 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896427 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896428 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896429 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:00 proc_lsst.multi:146 [INFO] found job 16896445 in provider checkpoint proc_lsst.multi INFO: found job 16896445 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 164788556c34cbf307f38618dfa3e9fd with result from task 2587 parsl.dataflow.dflow DEBUG: Task 2587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/fa9b1fb6-08a5-4cb0-93af-6976cb80cff8_isrForCrosstalkSources_1125906_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/fa9b1fb6-08a5-4cb0-93af-6976cb80cff8_isrForCrosstalkSources_1125906_18.stderr parsl.dataflow.dflow INFO: Task 2312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c037b6093bcce4bbba15a94d4d4e3e6a with result from task 2312 parsl.dataflow.dflow DEBUG: Task 2312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/707d562f-c4ae-4f2a-bd9a-e0214f1afa04_isrForCrosstalkSources_1125899_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/707d562f-c4ae-4f2a-bd9a-e0214f1afa04_isrForCrosstalkSources_1125899_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:00 proc_lsst.multi:146 [INFO] found job 16896446 in provider checkpoint proc_lsst.multi INFO: found job 16896446 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da38a90291089274eb8ffb7825569f81 with result from task 2673 parsl.dataflow.dflow DEBUG: Task 2673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/80c968c4-0ea3-496e-9167-287c81a3eb04_isrForCrosstalkSources_1125926_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/80c968c4-0ea3-496e-9167-287c81a3eb04_isrForCrosstalkSources_1125926_22.stderr parsl.dataflow.dflow INFO: Task 2581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3089ab3bb939b9703c8a87fe40c46c16 with result from task 2581 parsl.dataflow.dflow DEBUG: Task 2581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/71e783a2-2d18-4dd4-a973-7672c126a6f2_isrForCrosstalkSources_1125907_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/71e783a2-2d18-4dd4-a973-7672c126a6f2_isrForCrosstalkSources_1125907_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:00 proc_lsst.multi:146 [INFO] found job 16896447 in provider checkpoint proc_lsst.multi INFO: found job 16896447 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4be93959808b04dce9794d497929ec8a with result from task 2561 parsl.dataflow.dflow DEBUG: Task 2561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/5b78bc60-fb05-4f8d-a209-9a54726f1905_isrForCrosstalkSources_1125906_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/5b78bc60-fb05-4f8d-a209-9a54726f1905_isrForCrosstalkSources_1125906_6.stderr parsl.dataflow.dflow INFO: Task 2672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf538658b61da0f7f5bd6428c7030d6b with result from task 2672 parsl.dataflow.dflow DEBUG: Task 2672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/0a0ba096-495d-42d1-931f-6ebde9eb2004_isrForCrosstalkSources_1125898_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/0a0ba096-495d-42d1-931f-6ebde9eb2004_isrForCrosstalkSources_1125898_8.stderr parsl.dataflow.dflow INFO: Task 2634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b39706dcf2eb942b91baa77aae67907 with result from task 2634 parsl.dataflow.dflow DEBUG: Task 2634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/1d7e21a8-6acb-4cbc-94ca-e6c8d37df03f_isrForCrosstalkSources_1125889_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/1d7e21a8-6acb-4cbc-94ca-e6c8d37df03f_isrForCrosstalkSources_1125889_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:00 proc_lsst.multi:146 [INFO] found job 16896448 in provider checkpoint proc_lsst.multi INFO: found job 16896448 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93e79437c1f5a4a8cad0114ad7f1ad0a with result from task 2621 parsl.dataflow.dflow DEBUG: Task 2621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/d23df871-85ee-41a9-9b77-e5ea0eac087f_isrForCrosstalkSources_1125897_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/d23df871-85ee-41a9-9b77-e5ea0eac087f_isrForCrosstalkSources_1125897_51.stderr parsl.dataflow.dflow INFO: Task 2685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 403560196897b13d1c84ac50b9d88cc9 with result from task 2685 parsl.dataflow.dflow DEBUG: Task 2685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/76083684-89aa-4576-a31a-b2c95988aadd_isrForCrosstalkSources_1125889_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/76083684-89aa-4576-a31a-b2c95988aadd_isrForCrosstalkSources_1125889_46.stderr parsl.dataflow.dflow INFO: Task 2630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ccd2b474bfa0f58f4f99deaeea5ef58 with result from task 2630 parsl.dataflow.dflow DEBUG: Task 2630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/10715943-520b-445f-b0b4-17cdf4e61320_isrForCrosstalkSources_1125899_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/10715943-520b-445f-b0b4-17cdf4e61320_isrForCrosstalkSources_1125899_5.stderr parsl.dataflow.dflow INFO: Task 2708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cc9b51a03d39a455ae983fb941ff689 with result from task 2708 parsl.dataflow.dflow DEBUG: Task 2708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/62922134-24e7-452e-b794-876194066d59_isrForCrosstalkSources_1125915_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/62922134-24e7-452e-b794-876194066d59_isrForCrosstalkSources_1125915_40.stderr parsl.dataflow.dflow INFO: Task 2668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06207e1a43bc85343c021b8ea7f7f8a7 with result from task 2668 parsl.dataflow.dflow DEBUG: Task 2668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/d5828e4d-66e4-4a94-b45b-fb4fb9ee81ac_isrForCrosstalkSources_1125928_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/d5828e4d-66e4-4a94-b45b-fb4fb9ee81ac_isrForCrosstalkSources_1125928_5.stderr parsl.dataflow.dflow INFO: Task 2596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2baef57d1077afe2ae7c6f386be7d27 with result from task 2596 parsl.dataflow.dflow DEBUG: Task 2596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/e05f0228-1bac-4f20-8180-28f93bc31c88_isrForCrosstalkSources_1125910_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/e05f0228-1bac-4f20-8180-28f93bc31c88_isrForCrosstalkSources_1125910_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:00 proc_lsst.multi:146 [INFO] found job 16896449 in provider checkpoint proc_lsst.multi INFO: found job 16896449 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4b7cfdf197647d06d3435d2b6e08ca6 with result from task 2559 parsl.dataflow.dflow DEBUG: Task 2559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/fb724fb9-9ebd-43f3-8e53-bdcb9e7195aa_isrForCrosstalkSources_1125944_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/fb724fb9-9ebd-43f3-8e53-bdcb9e7195aa_isrForCrosstalkSources_1125944_35.stderr parsl.dataflow.dflow INFO: Task 2691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14d76b0d52621ee9ed01741cbf1fcf69 with result from task 2691 parsl.dataflow.dflow DEBUG: Task 2691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/0fcd87db-0754-427d-9b20-2195d820c2d0_isrForCrosstalkSources_1125891_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/0fcd87db-0754-427d-9b20-2195d820c2d0_isrForCrosstalkSources_1125891_23.stderr parsl.dataflow.dflow INFO: Task 2607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b7093fc7fcf3a507b8f917ad9b0016a with result from task 2607 parsl.dataflow.dflow DEBUG: Task 2607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/2f1d2b29-c548-45cf-ba36-3284a027075e_isrForCrosstalkSources_1125899_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/2f1d2b29-c548-45cf-ba36-3284a027075e_isrForCrosstalkSources_1125899_50.stderr parsl.dataflow.dflow INFO: Task 2642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e000df5f47c8daeaf3e39cd21dbe484 with result from task 2642 parsl.dataflow.dflow DEBUG: Task 2642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/200f23f1-7c27-45d7-9843-ee6575022775_isrForCrosstalkSources_1125910_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/200f23f1-7c27-45d7-9843-ee6575022775_isrForCrosstalkSources_1125910_58.stderr parsl.dataflow.dflow INFO: Task 2606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e676bf547592995a79a1a95ab4c245e with result from task 2606 parsl.dataflow.dflow DEBUG: Task 2606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/cecc248a-c133-4b5d-a586-13422a1fbac7_isrForCrosstalkSources_1125889_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/cecc248a-c133-4b5d-a586-13422a1fbac7_isrForCrosstalkSources_1125889_12.stderr parsl.dataflow.dflow INFO: Task 2662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d39f44926355b3c25b5116996e89f83e with result from task 2662 parsl.dataflow.dflow DEBUG: Task 2662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/78887e37-cbb9-48a9-9252-ece739ab110f_isrForCrosstalkSources_1125917_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/78887e37-cbb9-48a9-9252-ece739ab110f_isrForCrosstalkSources_1125917_50.stderr parsl.dataflow.dflow INFO: Task 2633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af6b343fcf7963c36f6d2b324b1d3529 with result from task 2633 parsl.dataflow.dflow DEBUG: Task 2633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/71bd08f7-02c2-4e09-bdc5-298733ca65bf_isrForCrosstalkSources_1125917_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/71bd08f7-02c2-4e09-bdc5-298733ca65bf_isrForCrosstalkSources_1125917_42.stderr parsl.dataflow.dflow INFO: Task 2569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50cea363ca1ebd3ad49eb36560deb865 with result from task 2569 parsl.dataflow.dflow DEBUG: Task 2569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/2689b0f5-286d-4333-981e-21b673badbf7_isrForCrosstalkSources_1125935_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/2689b0f5-286d-4333-981e-21b673badbf7_isrForCrosstalkSources_1125935_45.stderr parsl.dataflow.dflow INFO: Task 2652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 685792b34fdec63fa0e3b9a369a24972 with result from task 2652 parsl.dataflow.dflow DEBUG: Task 2652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/602089ad-50f1-45b3-aa35-7d53d998a8a5_isrForCrosstalkSources_1125931_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/602089ad-50f1-45b3-aa35-7d53d998a8a5_isrForCrosstalkSources_1125931_52.stderr parsl.dataflow.dflow INFO: Task 2568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdfe580d23fb818c9f065c2e6db9bf8d with result from task 2568 parsl.dataflow.dflow DEBUG: Task 2568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/633f78dd-9103-43fc-bf35-07895d35f59a_isrForCrosstalkSources_1125946_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/633f78dd-9103-43fc-bf35-07895d35f59a_isrForCrosstalkSources_1125946_44.stderr parsl.dataflow.dflow INFO: Task 2649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5291f7b8c837625c63c960644b35ab0d with result from task 2649 parsl.dataflow.dflow DEBUG: Task 2649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/5d26cb69-64b9-40c1-83ee-ff6fcf1d502e_isrForCrosstalkSources_1125912_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/5d26cb69-64b9-40c1-83ee-ff6fcf1d502e_isrForCrosstalkSources_1125912_58.stderr parsl.dataflow.dflow INFO: Task 2643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb9df3de9355be82db882453470bd4b2 with result from task 2643 parsl.dataflow.dflow DEBUG: Task 2643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/18efb0da-f470-49e0-a386-66d44fed0733_isrForCrosstalkSources_1125898_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/18efb0da-f470-49e0-a386-66d44fed0733_isrForCrosstalkSources_1125898_51.stderr parsl.dataflow.dflow INFO: Task 2580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd47436391be544990792892fd42c19a with result from task 2580 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:01 proc_lsst.multi:146 [INFO] found job 16896450 in provider checkpoint proc_lsst.multi INFO: found job 16896450 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/a9419541-1396-469e-b8be-b844bb916090_isrForCrosstalkSources_1125923_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/a9419541-1396-469e-b8be-b844bb916090_isrForCrosstalkSources_1125923_38.stderr parsl.dataflow.dflow INFO: Task 2620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 466aa227796e36209247cf8c6a060521 with result from task 2620 parsl.dataflow.dflow DEBUG: Task 2620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/7dc68754-8277-4234-9cf7-f84d1356c43a_isrForCrosstalkSources_1125903_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/7dc68754-8277-4234-9cf7-f84d1356c43a_isrForCrosstalkSources_1125903_7.stderr parsl.dataflow.dflow INFO: Task 2664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7589885573be367721c3e09759094e83 with result from task 2664 parsl.dataflow.dflow DEBUG: Task 2664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/aa5164a1-721d-4d42-bffa-1e798bef8c43_isrForCrosstalkSources_1125934_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/aa5164a1-721d-4d42-bffa-1e798bef8c43_isrForCrosstalkSources_1125934_39.stderr parsl.dataflow.dflow INFO: Task 2598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfaed8476dfb82c507a3942028c09de3 with result from task 2598 parsl.dataflow.dflow DEBUG: Task 2598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/80d408d3-1aec-4801-bbe0-2dcff0dcd59d_isrForCrosstalkSources_1125913_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/80d408d3-1aec-4801-bbe0-2dcff0dcd59d_isrForCrosstalkSources_1125913_54.stderr parsl.dataflow.dflow INFO: Task 2583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09aae4996027b84c827ff90144d67fff with result from task 2583 parsl.dataflow.dflow DEBUG: Task 2583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/c5c09f1f-5a5d-4074-b40e-6548b89f91df_isrForCrosstalkSources_1125913_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/c5c09f1f-5a5d-4074-b40e-6548b89f91df_isrForCrosstalkSources_1125913_50.stderr parsl.dataflow.dflow INFO: Task 2589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17c3fb3b1845503e5d1c38e9038b3999 with result from task 2589 parsl.dataflow.dflow DEBUG: Task 2589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e34d427a-eae8-43c8-9638-f34b7b877741_isrForCrosstalkSources_1125937_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e34d427a-eae8-43c8-9638-f34b7b877741_isrForCrosstalkSources_1125937_43.stderr parsl.dataflow.dflow INFO: Task 2683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7abb39c31988e9d10bac35f78e553a4 with result from task 2683 parsl.dataflow.dflow DEBUG: Task 2683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/8f1a353c-e40a-4425-9e08-8fdd9c99b55a_isrForCrosstalkSources_1125947_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/8f1a353c-e40a-4425-9e08-8fdd9c99b55a_isrForCrosstalkSources_1125947_47.stderr parsl.dataflow.dflow INFO: Task 2661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4cdccdd20dce537098211273d7012fc with result from task 2661 parsl.dataflow.dflow DEBUG: Task 2661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6cced87c-8688-4cd5-957d-a66b38896fae_isrForCrosstalkSources_1125909_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6cced87c-8688-4cd5-957d-a66b38896fae_isrForCrosstalkSources_1125909_21.stderr parsl.dataflow.dflow INFO: Task 2601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dd296f837fb46bec61910df87bc23cd with result from task 2601 parsl.dataflow.dflow DEBUG: Task 2601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/19acad07-e959-48c8-92f6-74e10a178f8b_isrForCrosstalkSources_1125919_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/19acad07-e959-48c8-92f6-74e10a178f8b_isrForCrosstalkSources_1125919_18.stderr parsl.dataflow.dflow INFO: Task 2575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5dfe8c3b88bb40fcbe601dc1038591e with result from task 2575 parsl.dataflow.dflow DEBUG: Task 2575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/e1f58907-8a96-4aa2-8f71-f75092c94059_isrForCrosstalkSources_1125903_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/e1f58907-8a96-4aa2-8f71-f75092c94059_isrForCrosstalkSources_1125903_28.stderr parsl.dataflow.dflow INFO: Task 2585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0bc3622270fb52db1f3ed2580f68226 with result from task 2585 parsl.dataflow.dflow DEBUG: Task 2585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6d4540de-93b2-4020-a185-58b84f185498_isrForCrosstalkSources_1125927_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6d4540de-93b2-4020-a185-58b84f185498_isrForCrosstalkSources_1125927_36.stderr parsl.dataflow.dflow INFO: Task 2692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 399630f1f13035f8164643ba33389e7b with result from task 2692 parsl.dataflow.dflow DEBUG: Task 2692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/47663458-4120-426f-aeca-5103152ca331_isrForCrosstalkSources_1125947_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/47663458-4120-426f-aeca-5103152ca331_isrForCrosstalkSources_1125947_45.stderr parsl.dataflow.dflow INFO: Task 2663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a034fe4f82902ad1f04de63b6cc30110 with result from task 2663 parsl.dataflow.dflow DEBUG: Task 2663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/54129f26-6071-47dc-8280-d0d5926e9784_isrForCrosstalkSources_1125933_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/54129f26-6071-47dc-8280-d0d5926e9784_isrForCrosstalkSources_1125933_24.stderr parsl.dataflow.dflow INFO: Task 2578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e3498cab5241a77ec78dec41361c6e8 with result from task 2578 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:01 proc_lsst.multi:146 [INFO] found job 16896451 in provider checkpoint proc_lsst.multi INFO: found job 16896451 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/b28378ae-e658-4748-aebd-230006eabb3c_isrForCrosstalkSources_1125905_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/b28378ae-e658-4748-aebd-230006eabb3c_isrForCrosstalkSources_1125905_1.stderr parsl.dataflow.dflow INFO: Task 2609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c13b50c8cc319477c324642a884245bc with result from task 2609 parsl.dataflow.dflow DEBUG: Task 2609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/c2bdf15b-e9b6-4643-a38e-81de43c9e39d_isrForCrosstalkSources_1125936_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/c2bdf15b-e9b6-4643-a38e-81de43c9e39d_isrForCrosstalkSources_1125936_11.stderr parsl.dataflow.dflow INFO: Task 2650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23519667df16c2deb663f2d9621d10d5 with result from task 2650 parsl.dataflow.dflow DEBUG: Task 2650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/152f8d47-d4fe-4882-9835-147a730bd23a_isrForCrosstalkSources_1125928_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/152f8d47-d4fe-4882-9835-147a730bd23a_isrForCrosstalkSources_1125928_3.stderr parsl.dataflow.dflow INFO: Task 2714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfb14291e9ff7c87ff722b940ffeae57 with result from task 2714 parsl.dataflow.dflow DEBUG: Task 2714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/abfd5eae-ecf4-4ae6-9272-6f94c5779f94_isrForCrosstalkSources_1125908_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/abfd5eae-ecf4-4ae6-9272-6f94c5779f94_isrForCrosstalkSources_1125908_19.stderr parsl.dataflow.dflow INFO: Task 2648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d16027d98958e09b2a9e12456e940e7a with result from task 2648 parsl.dataflow.dflow DEBUG: Task 2648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d718149f-f134-4c06-9cab-eec31ed795e6_isrForCrosstalkSources_1125940_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d718149f-f134-4c06-9cab-eec31ed795e6_isrForCrosstalkSources_1125940_36.stderr parsl.dataflow.dflow INFO: Task 2656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f36f1ec10f221abbff0bf9c80dec04e with result from task 2656 parsl.dataflow.dflow DEBUG: Task 2656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/677d2168-ea92-441e-9287-7358ea1e355b_isrForCrosstalkSources_1125910_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/677d2168-ea92-441e-9287-7358ea1e355b_isrForCrosstalkSources_1125910_26.stderr parsl.dataflow.dflow INFO: Task 2565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 543aab3c9d7b35741bea576853d9fdb1 with result from task 2565 parsl.dataflow.dflow DEBUG: Task 2565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/8827f50f-d79b-41ea-8956-29287a67a619_isrForCrosstalkSources_1125943_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/8827f50f-d79b-41ea-8956-29287a67a619_isrForCrosstalkSources_1125943_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:02 proc_lsst.multi:146 [INFO] found job 16896452 in provider checkpoint proc_lsst.multi INFO: found job 16896452 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a3677275765676507390eba7adebac4 with result from task 2571 parsl.dataflow.dflow DEBUG: Task 2571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/f84beaf5-dcb0-48b4-9657-de426099afbe_isrForCrosstalkSources_1125940_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/f84beaf5-dcb0-48b4-9657-de426099afbe_isrForCrosstalkSources_1125940_62.stderr parsl.dataflow.dflow INFO: Task 2599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 971bb8b1339f32bd4a18e99e9899f393 with result from task 2599 parsl.dataflow.dflow DEBUG: Task 2599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/41e15ebf-2e37-4319-9994-47770a91dcf1_isrForCrosstalkSources_1125943_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/41e15ebf-2e37-4319-9994-47770a91dcf1_isrForCrosstalkSources_1125943_9.stderr parsl.dataflow.dflow INFO: Task 2681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d42e0f04b889218cc7aef0c561d4f13 with result from task 2681 parsl.dataflow.dflow DEBUG: Task 2681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/75222876-dcb8-4bed-af33-18dfdf633718_isrForCrosstalkSources_1125934_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/75222876-dcb8-4bed-af33-18dfdf633718_isrForCrosstalkSources_1125934_50.stderr parsl.dataflow.dflow INFO: Task 2723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98009721a7a0c25b3e16d10f49a18512 with result from task 2723 parsl.dataflow.dflow DEBUG: Task 2723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/dff3f5ce-4223-4a84-b0ae-7575f65722ac_isrForCrosstalkSources_1125906_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/dff3f5ce-4223-4a84-b0ae-7575f65722ac_isrForCrosstalkSources_1125906_1.stderr parsl.dataflow.dflow INFO: Task 2665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4848e99f7744ee28824687bccc488b98 with result from task 2665 parsl.dataflow.dflow DEBUG: Task 2665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/fe63a309-6980-4a8d-80bc-40d108dfc1db_isrForCrosstalkSources_1125904_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/fe63a309-6980-4a8d-80bc-40d108dfc1db_isrForCrosstalkSources_1125904_32.stderr parsl.dataflow.dflow INFO: Task 2707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f02389b1f5a91e5cb1493e82493bde96 with result from task 2707 parsl.dataflow.dflow DEBUG: Task 2707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/bd296f32-8423-4daf-a0c6-3a9b35ae0936_isrForCrosstalkSources_1125907_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/bd296f32-8423-4daf-a0c6-3a9b35ae0936_isrForCrosstalkSources_1125907_33.stderr parsl.dataflow.dflow INFO: Task 2576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31707c9c8ed829284fd144639a5e92f2 with result from task 2576 parsl.dataflow.dflow DEBUG: Task 2576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/f44adea6-ae9b-4181-a970-b54ea8192961_isrForCrosstalkSources_1125907_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/f44adea6-ae9b-4181-a970-b54ea8192961_isrForCrosstalkSources_1125907_40.stderr parsl.dataflow.dflow INFO: Task 2724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09f8df738923a9e459605d99dd8624d1 with result from task 2724 parsl.dataflow.dflow DEBUG: Task 2724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/796a8d10-f1b1-4e8e-9d21-7e7ba5fce27c_isrForCrosstalkSources_1125927_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/796a8d10-f1b1-4e8e-9d21-7e7ba5fce27c_isrForCrosstalkSources_1125927_10.stderr parsl.dataflow.dflow INFO: Task 2579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8a3aef00673944bc797d37022fb5880 with result from task 2579 parsl.dataflow.dflow DEBUG: Task 2579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/03e4f45c-7794-41a0-a601-85536602a8aa_isrForCrosstalkSources_1125943_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/03e4f45c-7794-41a0-a601-85536602a8aa_isrForCrosstalkSources_1125943_35.stderr parsl.dataflow.dflow INFO: Task 2659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bbb7694ac35e35961df572ce348f5ba with result from task 2659 parsl.dataflow.dflow DEBUG: Task 2659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/844986aa-a028-4ab6-bbe3-54d7ec354097_isrForCrosstalkSources_1125913_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/844986aa-a028-4ab6-bbe3-54d7ec354097_isrForCrosstalkSources_1125913_28.stderr parsl.dataflow.dflow INFO: Task 2713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5c27dadff8ae4940f7e56d1b06cc78f with result from task 2713 parsl.dataflow.dflow DEBUG: Task 2713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/9d486282-6b5c-4be2-862e-075d85cc77b6_isrForCrosstalkSources_1125888_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/9d486282-6b5c-4be2-862e-075d85cc77b6_isrForCrosstalkSources_1125888_5.stderr parsl.dataflow.dflow INFO: Task 2600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ce9c833d0b8a1572df502358a26c6ed with result from task 2600 parsl.dataflow.dflow DEBUG: Task 2600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/0b48e4ee-bde2-4985-94c7-b29b7ea669a2_isrForCrosstalkSources_1125933_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/0b48e4ee-bde2-4985-94c7-b29b7ea669a2_isrForCrosstalkSources_1125933_39.stderr parsl.dataflow.dflow INFO: Task 2705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a7b304ce010b0b2ed23cd5dd4655b81 with result from task 2705 parsl.dataflow.dflow DEBUG: Task 2705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/7df485ed-834a-4f2e-9493-507088e09d4b_isrForCrosstalkSources_1125915_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/7df485ed-834a-4f2e-9493-507088e09d4b_isrForCrosstalkSources_1125915_61.stderr parsl.dataflow.dflow INFO: Task 2684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b51a9af396b5888d46b69d940312617b with result from task 2684 parsl.dataflow.dflow DEBUG: Task 2684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/1c5b1583-1f61-4e07-8410-3d84d0b3266a_isrForCrosstalkSources_1125938_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/1c5b1583-1f61-4e07-8410-3d84d0b3266a_isrForCrosstalkSources_1125938_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:02 proc_lsst.multi:146 [INFO] found job 16896453 in provider checkpoint proc_lsst.multi INFO: found job 16896453 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbf2dd88e0108cc461b8413f073c1041 with result from task 2588 parsl.dataflow.dflow DEBUG: Task 2588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/28ef3fa2-4e4f-4fe1-a286-aa50b1d896a5_isrForCrosstalkSources_1125900_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/28ef3fa2-4e4f-4fe1-a286-aa50b1d896a5_isrForCrosstalkSources_1125900_25.stderr parsl.dataflow.dflow INFO: Task 2639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66b7f33b7efd04a75f3eca6d71c90a19 with result from task 2639 parsl.dataflow.dflow DEBUG: Task 2639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/1dd39909-b27d-4971-b8f4-f3521f1eeaa5_isrForCrosstalkSources_1125899_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/1dd39909-b27d-4971-b8f4-f3521f1eeaa5_isrForCrosstalkSources_1125899_48.stderr parsl.dataflow.dflow INFO: Task 2635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0e8ad2af913f1747960767347e56ebb with result from task 2635 parsl.dataflow.dflow DEBUG: Task 2635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/3808e67a-9e70-4a19-a97c-20df76fb6558_isrForCrosstalkSources_1125925_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/3808e67a-9e70-4a19-a97c-20df76fb6558_isrForCrosstalkSources_1125925_16.stderr parsl.dataflow.dflow INFO: Task 2720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd619e119c32f021f5458305764e8e41 with result from task 2720 parsl.dataflow.dflow DEBUG: Task 2720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/1b0e4209-4d90-48be-95c7-aa469d69710b_isrForCrosstalkSources_1125928_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/1b0e4209-4d90-48be-95c7-aa469d69710b_isrForCrosstalkSources_1125928_38.stderr parsl.dataflow.dflow INFO: Task 2695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cae42d4946081855dbe5b0551192bacb with result from task 2695 parsl.dataflow.dflow DEBUG: Task 2695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/ceb601ad-0674-4c60-baf7-f06e38891396_isrForCrosstalkSources_1125919_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/ceb601ad-0674-4c60-baf7-f06e38891396_isrForCrosstalkSources_1125919_7.stderr parsl.dataflow.dflow INFO: Task 2644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae31c9ffba05d4e824443524a44a299a with result from task 2644 parsl.dataflow.dflow DEBUG: Task 2644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/338ceecf-4c2e-4ccd-a54c-0515f7bb2fca_isrForCrosstalkSources_1125901_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/338ceecf-4c2e-4ccd-a54c-0515f7bb2fca_isrForCrosstalkSources_1125901_37.stderr parsl.dataflow.dflow INFO: Task 2722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f8b16f6a655b2444c37fb97a273668a with result from task 2722 parsl.dataflow.dflow DEBUG: Task 2722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/1d472ed3-0fed-434d-b57a-4ed0c79d7c95_isrForCrosstalkSources_1125902_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/1d472ed3-0fed-434d-b57a-4ed0c79d7c95_isrForCrosstalkSources_1125902_39.stderr parsl.dataflow.dflow INFO: Task 2754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d49e6c5b937e4e517e473e7a9c9a1efa with result from task 2754 parsl.dataflow.dflow DEBUG: Task 2754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/25ea66ea-835b-4fdf-9980-dcb89545f870_isrForCrosstalkSources_1125910_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/25ea66ea-835b-4fdf-9980-dcb89545f870_isrForCrosstalkSources_1125910_22.stderr parsl.dataflow.dflow INFO: Task 2608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbc1e7b43193e09e0c5b45361520138f with result from task 2608 parsl.dataflow.dflow DEBUG: Task 2608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c55b1c89-5db8-44d0-826a-816c1dceef34_isrForCrosstalkSources_1125910_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c55b1c89-5db8-44d0-826a-816c1dceef34_isrForCrosstalkSources_1125910_18.stderr parsl.dataflow.dflow INFO: Task 2562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8640b58d843571300204d90b0f0f8115 with result from task 2562 parsl.dataflow.dflow DEBUG: Task 2562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2ebbdc4b-c2a3-4da9-9d41-29d6398c682d_isrForCrosstalkSources_1125898_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2ebbdc4b-c2a3-4da9-9d41-29d6398c682d_isrForCrosstalkSources_1125898_52.stderr parsl.dataflow.dflow INFO: Task 2628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c78a158b5a03ca0d0d90d2a25f21ea8c with result from task 2628 parsl.dataflow.dflow DEBUG: Task 2628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/7c3ec2f7-788a-45f4-99c9-07fabeafc659_isrForCrosstalkSources_1125933_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/7c3ec2f7-788a-45f4-99c9-07fabeafc659_isrForCrosstalkSources_1125933_27.stderr parsl.dataflow.dflow INFO: Task 2611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1492f6e63631de2e703d0d40c78cac5 with result from task 2611 parsl.dataflow.dflow DEBUG: Task 2611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/0401ea02-1dd7-4509-a05f-b41db826ad9e_isrForCrosstalkSources_1125933_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/0401ea02-1dd7-4509-a05f-b41db826ad9e_isrForCrosstalkSources_1125933_26.stderr parsl.dataflow.dflow INFO: Task 2658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f3601826092331db0f6fd2d33a8cdab with result from task 2658 parsl.dataflow.dflow DEBUG: Task 2658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/8774bc01-e0d8-4bb3-9ebd-b8dd643878ad_isrForCrosstalkSources_1125904_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/8774bc01-e0d8-4bb3-9ebd-b8dd643878ad_isrForCrosstalkSources_1125904_25.stderr parsl.dataflow.dflow INFO: Task 2623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e4114b93cb11c67de00cb674d7830cb with result from task 2623 parsl.dataflow.dflow DEBUG: Task 2623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/0999f154-d663-4f3f-b174-9490c3a6fc11_isrForCrosstalkSources_1125888_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/0999f154-d663-4f3f-b174-9490c3a6fc11_isrForCrosstalkSources_1125888_37.stderr parsl.dataflow.dflow INFO: Task 2625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af0d6dae07314df226ca5b9d7d43ae2a with result from task 2625 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:03 proc_lsst.multi:146 [INFO] found job 16896454 in provider checkpoint proc_lsst.multi INFO: found job 16896454 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/81917196-c0ce-433c-ab56-cb0115073c15_isrForCrosstalkSources_1125923_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/81917196-c0ce-433c-ab56-cb0115073c15_isrForCrosstalkSources_1125923_17.stderr parsl.dataflow.dflow INFO: Task 2560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c631fd7b5c1a1e038e50c7aeff5fe25 with result from task 2560 parsl.dataflow.dflow DEBUG: Task 2560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/198e3209-0585-4c6d-bd96-ab9b77983213_isrForCrosstalkSources_1125931_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/198e3209-0585-4c6d-bd96-ab9b77983213_isrForCrosstalkSources_1125931_42.stderr parsl.dataflow.dflow INFO: Task 2675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0af8248d4971079ab57e04b9a959d403 with result from task 2675 parsl.dataflow.dflow DEBUG: Task 2675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/fa149316-25e5-4fb4-8e4c-284c162858b8_isrForCrosstalkSources_1125910_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/fa149316-25e5-4fb4-8e4c-284c162858b8_isrForCrosstalkSources_1125910_39.stderr parsl.dataflow.dflow INFO: Task 2814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dbbf9b3640b7f8ce15f60e2646371f1 with result from task 2814 parsl.dataflow.dflow DEBUG: Task 2814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/24bd5bed-b62a-4aa4-8082-9a5e5103e2e1_isrForCrosstalkSources_1125916_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/24bd5bed-b62a-4aa4-8082-9a5e5103e2e1_isrForCrosstalkSources_1125916_13.stderr parsl.dataflow.dflow INFO: Task 2711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b3cc94eb0589fcb2077048794215f66 with result from task 2711 parsl.dataflow.dflow DEBUG: Task 2711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e497055b-6510-4778-8db7-e27589fe7aef_isrForCrosstalkSources_1125916_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e497055b-6510-4778-8db7-e27589fe7aef_isrForCrosstalkSources_1125916_19.stderr parsl.dataflow.dflow INFO: Task 2615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49536804d63217c02618e8266cb98df3 with result from task 2615 parsl.dataflow.dflow DEBUG: Task 2615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/d290a5f4-2b13-46b0-82ca-1e37257f403b_isrForCrosstalkSources_1125946_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/d290a5f4-2b13-46b0-82ca-1e37257f403b_isrForCrosstalkSources_1125946_31.stderr parsl.dataflow.dflow INFO: Task 2657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1364b7bcfde5a230dd360fb961d5c3c with result from task 2657 parsl.dataflow.dflow DEBUG: Task 2657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/f23d6fec-350e-4c8a-84e1-8dfc87cad1da_isrForCrosstalkSources_1125893_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/f23d6fec-350e-4c8a-84e1-8dfc87cad1da_isrForCrosstalkSources_1125893_38.stderr parsl.dataflow.dflow INFO: Task 2755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9c1da9db8078d099c03fb17a6879388 with result from task 2755 parsl.dataflow.dflow DEBUG: Task 2755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/ff39a3c1-b884-4148-9426-d4079959ecf4_isrForCrosstalkSources_1125943_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/ff39a3c1-b884-4148-9426-d4079959ecf4_isrForCrosstalkSources_1125943_46.stderr parsl.dataflow.dflow INFO: Task 2646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11090402378d1869aa334c0dc75f7dda with result from task 2646 parsl.dataflow.dflow DEBUG: Task 2646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/d7bd947d-2f7c-40e4-a4d9-f14638c9ceec_isrForCrosstalkSources_1125929_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/d7bd947d-2f7c-40e4-a4d9-f14638c9ceec_isrForCrosstalkSources_1125929_13.stderr parsl.dataflow.dflow INFO: Task 2671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a3235eb9a21bda2fa029d9843d54507 with result from task 2671 parsl.dataflow.dflow DEBUG: Task 2671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/125496ec-3a41-4fb2-ba23-cbe4dec5d8f7_isrForCrosstalkSources_1125922_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/125496ec-3a41-4fb2-ba23-cbe4dec5d8f7_isrForCrosstalkSources_1125922_24.stderr parsl.dataflow.dflow INFO: Task 2614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64139d7f627c57df1716e64465cfcbe8 with result from task 2614 parsl.dataflow.dflow DEBUG: Task 2614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d62c1277-db3e-40da-97a9-a087192eeee7_isrForCrosstalkSources_1125923_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/d62c1277-db3e-40da-97a9-a087192eeee7_isrForCrosstalkSources_1125923_46.stderr parsl.dataflow.dflow INFO: Task 2617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4883fefed0755004efac0ce5006035a5 with result from task 2617 parsl.dataflow.dflow DEBUG: Task 2617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/31f1adfb-23b8-4da8-a904-fe1dbb0c68aa_isrForCrosstalkSources_1125942_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/31f1adfb-23b8-4da8-a904-fe1dbb0c68aa_isrForCrosstalkSources_1125942_36.stderr parsl.dataflow.dflow INFO: Task 2660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d2ccb5df92fcd0507addcbba2145acd with result from task 2660 parsl.dataflow.dflow DEBUG: Task 2660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/f4f11d09-3ce5-44d3-9643-7620d9f2cf99_isrForCrosstalkSources_1125901_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/f4f11d09-3ce5-44d3-9643-7620d9f2cf99_isrForCrosstalkSources_1125901_25.stderr parsl.dataflow.dflow INFO: Task 2700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 724bdc3bb03467c132f35b81a86c1c6b with result from task 2700 parsl.dataflow.dflow DEBUG: Task 2700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/c299e983-fefe-4d05-8e33-f8f318974e7d_isrForCrosstalkSources_1125941_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/c299e983-fefe-4d05-8e33-f8f318974e7d_isrForCrosstalkSources_1125941_16.stderr parsl.dataflow.dflow INFO: Task 2669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e48a421f4aa8a991ea4e833ef43f8546 with result from task 2669 parsl.dataflow.dflow DEBUG: Task 2669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/f5ddeb07-c8e1-4273-bf8e-fe1d89be22b5_isrForCrosstalkSources_1125921_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/f5ddeb07-c8e1-4273-bf8e-fe1d89be22b5_isrForCrosstalkSources_1125921_31.stderr parsl.dataflow.dflow INFO: Task 2758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d23bb0dc71440d4af72dfd3da713eab with result from task 2758 parsl.dataflow.dflow DEBUG: Task 2758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/2bb3120f-ebc8-49c9-8da7-f5232a14a92a_isrForCrosstalkSources_1125897_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/2bb3120f-ebc8-49c9-8da7-f5232a14a92a_isrForCrosstalkSources_1125897_2.stderr parsl.dataflow.dflow INFO: Task 2820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f64f020e9b09dde135d7c71d9bd05512 with result from task 2820 parsl.dataflow.dflow DEBUG: Task 2820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/877618ef-44d7-4a1d-a991-5682b1e1c706_isrForCrosstalkSources_1125899_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/877618ef-44d7-4a1d-a991-5682b1e1c706_isrForCrosstalkSources_1125899_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:03 proc_lsst.multi:146 [INFO] found job 16896455 in provider checkpoint proc_lsst.multi INFO: found job 16896455 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 918b190e6604a655d27cf78f0a3b7a60 with result from task 2632 parsl.dataflow.dflow DEBUG: Task 2632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/905271c9-2e93-40e2-a675-399643d26eea_isrForCrosstalkSources_1125904_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/905271c9-2e93-40e2-a675-399643d26eea_isrForCrosstalkSources_1125904_19.stderr parsl.dataflow.dflow INFO: Task 2717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d3ddc06bd4fd3ae582480b8bed1b856 with result from task 2717 parsl.dataflow.dflow DEBUG: Task 2717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/18e8226f-7ce4-421a-bbe1-6e162acf2621_isrForCrosstalkSources_1125905_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/18e8226f-7ce4-421a-bbe1-6e162acf2621_isrForCrosstalkSources_1125905_20.stderr parsl.dataflow.dflow INFO: Task 2702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa0498ff51ac8a2fc672168885f5c38d with result from task 2702 parsl.dataflow.dflow DEBUG: Task 2702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/23f9f7d5-0714-47ba-9da9-a904d790b8fb_isrForCrosstalkSources_1125906_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/23f9f7d5-0714-47ba-9da9-a904d790b8fb_isrForCrosstalkSources_1125906_47.stderr parsl.dataflow.dflow INFO: Task 2616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7027edbc51dda96ba0dd573659adc570 with result from task 2616 parsl.dataflow.dflow DEBUG: Task 2616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/6fe25fdf-99f5-46e0-80d3-149635429eda_isrForCrosstalkSources_1125890_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/6fe25fdf-99f5-46e0-80d3-149635429eda_isrForCrosstalkSources_1125890_29.stderr parsl.dataflow.dflow INFO: Task 2784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5767b9d70637db88f8c1d94e2019603e with result from task 2784 parsl.dataflow.dflow DEBUG: Task 2784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/319c4c7e-d46e-400a-aebc-a31c2382b76e_isrForCrosstalkSources_1125920_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/319c4c7e-d46e-400a-aebc-a31c2382b76e_isrForCrosstalkSources_1125920_50.stderr parsl.dataflow.dflow INFO: Task 2641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0154df8b4f43d037ab65edb0e4b5f43b with result from task 2641 parsl.dataflow.dflow DEBUG: Task 2641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/0355559d-91c6-4655-bd54-5c1cd127038d_isrForCrosstalkSources_1125906_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/0355559d-91c6-4655-bd54-5c1cd127038d_isrForCrosstalkSources_1125906_39.stderr parsl.dataflow.dflow INFO: Task 2742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba6cc92b2f06798f17951a85dabcdbc8 with result from task 2742 parsl.dataflow.dflow DEBUG: Task 2742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/1bf96037-d0a0-40a3-9ac9-03730960522b_isrForCrosstalkSources_1125947_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/1bf96037-d0a0-40a3-9ac9-03730960522b_isrForCrosstalkSources_1125947_41.stderr parsl.dataflow.dflow INFO: Task 2636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21387d9ad0e57c0ca589f21371fad52b with result from task 2636 parsl.dataflow.dflow DEBUG: Task 2636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/fc7f1af3-170c-4705-981f-c10b65647dc9_isrForCrosstalkSources_1125902_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/fc7f1af3-170c-4705-981f-c10b65647dc9_isrForCrosstalkSources_1125902_56.stderr parsl.dataflow.dflow INFO: Task 2624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a80dd5f274adf07bec90905676857c79 with result from task 2624 parsl.dataflow.dflow DEBUG: Task 2624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/04a6ee66-e2ab-4732-ad37-119a53419c7d_isrForCrosstalkSources_1125915_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/04a6ee66-e2ab-4732-ad37-119a53419c7d_isrForCrosstalkSources_1125915_15.stderr parsl.dataflow.dflow INFO: Task 2718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bbbfb7532c89e289d65b2327f3b2089 with result from task 2718 parsl.dataflow.dflow DEBUG: Task 2718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/6ccb60e1-1647-488b-983c-39780af3928b_isrForCrosstalkSources_1125921_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/6ccb60e1-1647-488b-983c-39780af3928b_isrForCrosstalkSources_1125921_32.stderr parsl.dataflow.dflow INFO: Task 2726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5af3aa25294585c1d4f7310dc13cbde with result from task 2726 parsl.dataflow.dflow DEBUG: Task 2726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/1a424be0-5f4e-47ca-80c3-7ff80d7a38b6_isrForCrosstalkSources_1125940_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/1a424be0-5f4e-47ca-80c3-7ff80d7a38b6_isrForCrosstalkSources_1125940_2.stderr parsl.dataflow.dflow INFO: Task 2706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 495074b2439689c077b51c7098143555 with result from task 2706 parsl.dataflow.dflow DEBUG: Task 2706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/f71b4797-a4dd-41b4-8b02-d28a8f7ce8fe_isrForCrosstalkSources_1125915_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/f71b4797-a4dd-41b4-8b02-d28a8f7ce8fe_isrForCrosstalkSources_1125915_51.stderr parsl.dataflow.dflow INFO: Task 2759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b241bfd41a62d8f38013a23293cd26c3 with result from task 2759 parsl.dataflow.dflow DEBUG: Task 2759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/7386a5d1-2006-453c-99df-4b870d830d81_isrForCrosstalkSources_1125930_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/7386a5d1-2006-453c-99df-4b870d830d81_isrForCrosstalkSources_1125930_7.stderr parsl.dataflow.dflow INFO: Task 2745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33fb57eded4ddcebdb3fa906f069746f with result from task 2745 parsl.dataflow.dflow DEBUG: Task 2745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/c0607eb7-0267-4a80-a0a3-4b1699278326_isrForCrosstalkSources_1125898_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/c0607eb7-0267-4a80-a0a3-4b1699278326_isrForCrosstalkSources_1125898_31.stderr parsl.dataflow.dflow INFO: Task 2690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eafe8a9216474ca773e00410a506422c with result from task 2690 parsl.dataflow.dflow DEBUG: Task 2690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/62f99ea7-5386-42c3-ae39-487d49e3a082_isrForCrosstalkSources_1125938_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/62f99ea7-5386-42c3-ae39-487d49e3a082_isrForCrosstalkSources_1125938_17.stderr parsl.dataflow.dflow INFO: Task 2734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6679c979a12ffa03f9d7b2a6ccbc8a6 with result from task 2734 parsl.dataflow.dflow DEBUG: Task 2734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/4ba69243-cd00-4dfc-9148-1a6ba0930739_isrForCrosstalkSources_1125932_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/4ba69243-cd00-4dfc-9148-1a6ba0930739_isrForCrosstalkSources_1125932_46.stderr parsl.dataflow.dflow INFO: Task 2637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d613359a933a69fcc9fda153bf87f80 with result from task 2637 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:04 proc_lsst.multi:146 [INFO] found job 16896456 in provider checkpoint proc_lsst.multi INFO: found job 16896456 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/354aa9b4-46ba-437b-bcbe-f2e986a7ba4c_isrForCrosstalkSources_1125946_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/354aa9b4-46ba-437b-bcbe-f2e986a7ba4c_isrForCrosstalkSources_1125946_34.stderr parsl.dataflow.dflow INFO: Task 2640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98348b458a87065ca5a7527b3ecc73a7 with result from task 2640 parsl.dataflow.dflow DEBUG: Task 2640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/8cfd3d42-4d38-4cfb-ab1c-8879c8264e3b_isrForCrosstalkSources_1125941_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/8cfd3d42-4d38-4cfb-ab1c-8879c8264e3b_isrForCrosstalkSources_1125941_54.stderr parsl.dataflow.dflow INFO: Task 2655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef1463f95ba52f677d15c793c849d64f with result from task 2655 parsl.dataflow.dflow DEBUG: Task 2655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/404b9a87-0d86-42ee-8830-90b5b741fee4_isrForCrosstalkSources_1125946_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/404b9a87-0d86-42ee-8830-90b5b741fee4_isrForCrosstalkSources_1125946_62.stderr parsl.dataflow.dflow INFO: Task 2772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 339bbba20f6f3cb1625cbc55051a0c5e with result from task 2772 parsl.dataflow.dflow DEBUG: Task 2772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/5142e845-e6c7-421d-a8d3-46d0e516c6ad_isrForCrosstalkSources_1125903_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/5142e845-e6c7-421d-a8d3-46d0e516c6ad_isrForCrosstalkSources_1125903_8.stderr parsl.dataflow.dflow INFO: Task 2773 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 111a24fd456912818ca176d395ad8004 with result from task 2773 parsl.dataflow.dflow DEBUG: Task 2773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/9b8f1ddd-c493-4da5-b351-32c92f4f0a60_isrForCrosstalkSources_1125927_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/9b8f1ddd-c493-4da5-b351-32c92f4f0a60_isrForCrosstalkSources_1125927_31.stderr parsl.dataflow.dflow INFO: Task 2813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0057d8073d0e8c8fb47d6f50f471b7f with result from task 2813 parsl.dataflow.dflow DEBUG: Task 2813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2a3d9367-e296-4d5e-b65a-d9ae832d0d0e_isrForCrosstalkSources_1125895_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/2a3d9367-e296-4d5e-b65a-d9ae832d0d0e_isrForCrosstalkSources_1125895_42.stderr parsl.dataflow.dflow INFO: Task 2765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2135435e031f14f19fd387706febb53 with result from task 2765 parsl.dataflow.dflow DEBUG: Task 2765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/d21e0028-ed08-437e-8658-af5bbfa01b68_isrForCrosstalkSources_1125901_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/d21e0028-ed08-437e-8658-af5bbfa01b68_isrForCrosstalkSources_1125901_35.stderr parsl.dataflow.dflow INFO: Task 2696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd379f2e0511784e01646eb959aa1a4f with result from task 2696 parsl.dataflow.dflow DEBUG: Task 2696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/79de3fa9-1fd4-4ccc-9ba4-851c566ab04e_isrForCrosstalkSources_1125906_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/79de3fa9-1fd4-4ccc-9ba4-851c566ab04e_isrForCrosstalkSources_1125906_2.stderr parsl.dataflow.dflow INFO: Task 2674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b743617962f9ef3a67c0899defaf0ad3 with result from task 2674 parsl.dataflow.dflow DEBUG: Task 2674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/ee03c364-ae07-41b8-8085-f1cf73ea089a_isrForCrosstalkSources_1125944_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/ee03c364-ae07-41b8-8085-f1cf73ea089a_isrForCrosstalkSources_1125944_17.stderr parsl.dataflow.dflow INFO: Task 2728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e03b560bf19d8c9b067941ea34a2d9f0 with result from task 2728 parsl.dataflow.dflow DEBUG: Task 2728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/d1e398b5-6307-41ed-96ba-29991475c775_isrForCrosstalkSources_1125918_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/d1e398b5-6307-41ed-96ba-29991475c775_isrForCrosstalkSources_1125918_38.stderr parsl.dataflow.dflow INFO: Task 2798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7551f801457bcd5c430689c91e6a8793 with result from task 2798 parsl.dataflow.dflow DEBUG: Task 2798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/db1ff18b-89fc-417b-80ac-d22f2f5dabe1_isrForCrosstalkSources_1125891_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/db1ff18b-89fc-417b-80ac-d22f2f5dabe1_isrForCrosstalkSources_1125891_25.stderr parsl.dataflow.dflow INFO: Task 2762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc74bc9392fd5317ef57a9dc31d7c9ab with result from task 2762 parsl.dataflow.dflow DEBUG: Task 2762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/1a86f032-9b06-4fa5-a4c8-63225f209a5d_isrForCrosstalkSources_1125944_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/1a86f032-9b06-4fa5-a4c8-63225f209a5d_isrForCrosstalkSources_1125944_27.stderr parsl.dataflow.dflow INFO: Task 2771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4555a7b1327bdd72688fe58327c74721 with result from task 2771 parsl.dataflow.dflow DEBUG: Task 2771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/2b1d8841-0ba5-4c02-ae07-e0a5fd08d78d_isrForCrosstalkSources_1125932_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/2b1d8841-0ba5-4c02-ae07-e0a5fd08d78d_isrForCrosstalkSources_1125932_56.stderr parsl.dataflow.dflow INFO: Task 2710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b38e48fb3355537240584eb7dc64cd8 with result from task 2710 parsl.dataflow.dflow DEBUG: Task 2710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/417e8d4b-8176-424f-9304-3a662d63a45f_isrForCrosstalkSources_1125899_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/417e8d4b-8176-424f-9304-3a662d63a45f_isrForCrosstalkSources_1125899_60.stderr parsl.dataflow.dflow INFO: Task 2676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d954be0b61a721d96e04aff15d53d2e2 with result from task 2676 parsl.dataflow.dflow DEBUG: Task 2676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/e2c4e2b2-16dd-4e5c-9b45-6c1730e1837c_isrForCrosstalkSources_1125920_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/e2c4e2b2-16dd-4e5c-9b45-6c1730e1837c_isrForCrosstalkSources_1125920_18.stderr parsl.dataflow.dflow INFO: Task 2638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88cae02fe7f6a884bbc70920c8006efb with result from task 2638 parsl.dataflow.dflow DEBUG: Task 2638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/0601d9a0-d2e9-468c-af92-e8f3508221d3_isrForCrosstalkSources_1125929_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/0601d9a0-d2e9-468c-af92-e8f3508221d3_isrForCrosstalkSources_1125929_15.stderr parsl.dataflow.dflow INFO: Task 2716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb61cc9c84f091e903c405cf34e8de55 with result from task 2716 parsl.dataflow.dflow DEBUG: Task 2716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/dd0da4ee-0fcd-4968-b8ae-fe02989e6b7f_isrForCrosstalkSources_1125902_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/dd0da4ee-0fcd-4968-b8ae-fe02989e6b7f_isrForCrosstalkSources_1125902_33.stderr parsl.dataflow.dflow INFO: Task 2603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45f287e192636b1567b07eacd4d4db20 with result from task 2603 parsl.dataflow.dflow DEBUG: Task 2603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/7a2b785e-ab77-432d-a68a-e09a2a277c50_isrForCrosstalkSources_1125920_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/7a2b785e-ab77-432d-a68a-e09a2a277c50_isrForCrosstalkSources_1125920_46.stderr parsl.dataflow.dflow INFO: Task 2735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83c892339e9cbbe48a3717bca021be0d with result from task 2735 parsl.dataflow.dflow DEBUG: Task 2735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/abcc6bab-ad4b-4a38-a166-3011f92d2543_isrForCrosstalkSources_1125946_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/abcc6bab-ad4b-4a38-a166-3011f92d2543_isrForCrosstalkSources_1125946_20.stderr parsl.dataflow.dflow INFO: Task 2778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05db3533168588921dbfa68201533c86 with result from task 2778 parsl.dataflow.dflow DEBUG: Task 2778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/f6f8fc68-65a4-4920-a68f-17c069f3ce7c_isrForCrosstalkSources_1125920_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/f6f8fc68-65a4-4920-a68f-17c069f3ce7c_isrForCrosstalkSources_1125920_14.stderr parsl.dataflow.dflow INFO: Task 2803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c912d4fbfe19053beea715a4f474d06 with result from task 2803 parsl.dataflow.dflow DEBUG: Task 2803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/b306ac71-b40c-48f5-aa8f-14eaaf311028_isrForCrosstalkSources_1125891_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/b306ac71-b40c-48f5-aa8f-14eaaf311028_isrForCrosstalkSources_1125891_33.stderr parsl.dataflow.dflow INFO: Task 2741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f620c8f7d738fc410695603a5364c7fb with result from task 2741 parsl.dataflow.dflow DEBUG: Task 2741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/0b8fb144-2047-4dd2-baaa-b89dc6ec28b6_isrForCrosstalkSources_1125901_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/0b8fb144-2047-4dd2-baaa-b89dc6ec28b6_isrForCrosstalkSources_1125901_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:04 proc_lsst.multi:146 [INFO] found job 16896457 in provider checkpoint proc_lsst.multi INFO: found job 16896457 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fe629f775734cc3d07442db7a598092 with result from task 2699 parsl.dataflow.dflow DEBUG: Task 2699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/c6a23d5f-7e36-45d8-84dd-8cabf942c7e4_isrForCrosstalkSources_1125925_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/c6a23d5f-7e36-45d8-84dd-8cabf942c7e4_isrForCrosstalkSources_1125925_19.stderr parsl.dataflow.dflow INFO: Task 2666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68aed095f428a7982e95e590da507a00 with result from task 2666 parsl.dataflow.dflow DEBUG: Task 2666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/87a0764e-f347-4a78-81f3-a0351e5d22a2_isrForCrosstalkSources_1125918_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/87a0764e-f347-4a78-81f3-a0351e5d22a2_isrForCrosstalkSources_1125918_34.stderr parsl.dataflow.dflow INFO: Task 2631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f15cc5f556ad918b603bcd6ec99105a with result from task 2631 parsl.dataflow.dflow DEBUG: Task 2631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/b8ec21d3-7af7-4042-9018-d04bd807d3e0_isrForCrosstalkSources_1125914_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/b8ec21d3-7af7-4042-9018-d04bd807d3e0_isrForCrosstalkSources_1125914_52.stderr parsl.dataflow.dflow INFO: Task 2730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 292d47197f4787794a931f7d75312cf8 with result from task 2730 parsl.dataflow.dflow DEBUG: Task 2730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/dc09485a-a1bd-4d3d-bb8e-c89442d0ecfc_isrForCrosstalkSources_1125931_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/dc09485a-a1bd-4d3d-bb8e-c89442d0ecfc_isrForCrosstalkSources_1125931_27.stderr parsl.dataflow.dflow INFO: Task 2780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fe344bfcd978d73a592a670c13b8721 with result from task 2780 parsl.dataflow.dflow DEBUG: Task 2780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/aeaf92ec-cf53-45af-83f4-25607fd74a15_isrForCrosstalkSources_1125897_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/aeaf92ec-cf53-45af-83f4-25607fd74a15_isrForCrosstalkSources_1125897_1.stderr parsl.dataflow.dflow INFO: Task 2767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 528c04c5f99670d783a7c24bb81816ab with result from task 2767 parsl.dataflow.dflow DEBUG: Task 2767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/7f09cd96-63fd-4871-ad2b-d2dfa672d3e9_isrForCrosstalkSources_1125904_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/7f09cd96-63fd-4871-ad2b-d2dfa672d3e9_isrForCrosstalkSources_1125904_23.stderr parsl.dataflow.dflow INFO: Task 2744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07549bb201590e35a3d7c1fcd917cd61 with result from task 2744 parsl.dataflow.dflow DEBUG: Task 2744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/72a38c02-b489-4bec-920a-8fbacf0c57d7_isrForCrosstalkSources_1125902_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/72a38c02-b489-4bec-920a-8fbacf0c57d7_isrForCrosstalkSources_1125902_4.stderr parsl.dataflow.dflow INFO: Task 2693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62d848abc39e7e1f10aa2fb97884731c with result from task 2693 parsl.dataflow.dflow DEBUG: Task 2693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/39489c5d-9db3-4f98-a089-da9b0a3939f1_isrForCrosstalkSources_1125920_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/39489c5d-9db3-4f98-a089-da9b0a3939f1_isrForCrosstalkSources_1125920_12.stderr parsl.dataflow.dflow INFO: Task 2719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4898fa4beef36d572a5d20ebdbca5acf with result from task 2719 parsl.dataflow.dflow DEBUG: Task 2719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/b0c49bfe-8b59-476e-93ab-b90228dc9b64_isrForCrosstalkSources_1125898_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/b0c49bfe-8b59-476e-93ab-b90228dc9b64_isrForCrosstalkSources_1125898_43.stderr parsl.dataflow.dflow INFO: Task 2763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae0567856108e11eaa4b3568bc6dc96e with result from task 2763 parsl.dataflow.dflow DEBUG: Task 2763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/e775a41a-3b4f-449c-b0e2-12fba4d8b272_isrForCrosstalkSources_1125898_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/e775a41a-3b4f-449c-b0e2-12fba4d8b272_isrForCrosstalkSources_1125898_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:05 proc_lsst.multi:146 [INFO] found job 16896458 in provider checkpoint proc_lsst.multi INFO: found job 16896458 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a8d6f8d366c8a088af6ad6beae8951a with result from task 2802 parsl.dataflow.dflow DEBUG: Task 2802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/908fd2b4-023a-472e-8e9f-ce96539c9e67_isrForCrosstalkSources_1125937_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/908fd2b4-023a-472e-8e9f-ce96539c9e67_isrForCrosstalkSources_1125937_8.stderr parsl.dataflow.dflow INFO: Task 2785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c0b20c85405ffe207d71ba070bd4475 with result from task 2785 parsl.dataflow.dflow DEBUG: Task 2785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a2895cc3-aec8-4874-88f1-b77712e97405_isrForCrosstalkSources_1125900_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a2895cc3-aec8-4874-88f1-b77712e97405_isrForCrosstalkSources_1125900_48.stderr parsl.dataflow.dflow INFO: Task 2715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c92b1e8d949d10e39a2092119d0a9a8e with result from task 2715 parsl.dataflow.dflow DEBUG: Task 2715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/4819c1c7-9db2-4e23-820a-42234357c5c6_isrForCrosstalkSources_1125903_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/4819c1c7-9db2-4e23-820a-42234357c5c6_isrForCrosstalkSources_1125903_51.stderr parsl.dataflow.dflow INFO: Task 2701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a9a0b4b4cd357883d7303871843a397 with result from task 2701 parsl.dataflow.dflow DEBUG: Task 2701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/22d75827-d8b1-4b68-8e32-1ff2708c7b8c_isrForCrosstalkSources_1125892_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/22d75827-d8b1-4b68-8e32-1ff2708c7b8c_isrForCrosstalkSources_1125892_4.stderr parsl.dataflow.dflow INFO: Task 2703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df5502ed8e99e434f9ed3ee614c3c998 with result from task 2703 parsl.dataflow.dflow DEBUG: Task 2703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d18053ea-9adb-48f1-958e-58d90603cf1c_isrForCrosstalkSources_1125934_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/d18053ea-9adb-48f1-958e-58d90603cf1c_isrForCrosstalkSources_1125934_16.stderr parsl.dataflow.dflow INFO: Task 2777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 173c0bfaed0e417f79fb5078bc54ffea with result from task 2777 parsl.dataflow.dflow DEBUG: Task 2777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/27824355-e134-450b-b0ea-9bd944876f56_isrForCrosstalkSources_1125907_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/27824355-e134-450b-b0ea-9bd944876f56_isrForCrosstalkSources_1125907_61.stderr parsl.dataflow.dflow INFO: Task 2613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b903e9363bfac0e5557a3b2f99f7d972 with result from task 2613 parsl.dataflow.dflow DEBUG: Task 2613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ac21b8de-fdc4-4d09-b773-897cfddced79_isrForCrosstalkSources_1125894_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/ac21b8de-fdc4-4d09-b773-897cfddced79_isrForCrosstalkSources_1125894_38.stderr parsl.dataflow.dflow INFO: Task 2653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e839246eaff62bc5808c4b20661b9eba with result from task 2653 parsl.dataflow.dflow DEBUG: Task 2653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/1305bab8-5379-4bc0-b2e4-2ba893ce9851_isrForCrosstalkSources_1125922_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/1305bab8-5379-4bc0-b2e4-2ba893ce9851_isrForCrosstalkSources_1125922_54.stderr parsl.dataflow.dflow INFO: Task 2618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d056b77f480f2ddc3418b906b6f1a8db with result from task 2618 parsl.dataflow.dflow DEBUG: Task 2618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/5ad8d4c4-2a84-4721-a22b-36401a7d7cd8_isrForCrosstalkSources_1125916_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/5ad8d4c4-2a84-4721-a22b-36401a7d7cd8_isrForCrosstalkSources_1125916_1.stderr parsl.dataflow.dflow INFO: Task 2770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b8ea6024a8b6503fb950ce3ab70a096 with result from task 2770 parsl.dataflow.dflow DEBUG: Task 2770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/e43ca15d-4859-42da-b85e-10c4fb8ca093_isrForCrosstalkSources_1125888_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/e43ca15d-4859-42da-b85e-10c4fb8ca093_isrForCrosstalkSources_1125888_32.stderr parsl.dataflow.dflow INFO: Task 2793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f07c10c64e986a2b15a0754e775f2e16 with result from task 2793 parsl.dataflow.dflow DEBUG: Task 2793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/e10c35c3-53af-4cb8-973d-ff5b9dd871eb_isrForCrosstalkSources_1125946_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/e10c35c3-53af-4cb8-973d-ff5b9dd871eb_isrForCrosstalkSources_1125946_41.stderr parsl.dataflow.dflow INFO: Task 2733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ea698e5c0b5b4a00831c9472e63324e with result from task 2733 parsl.dataflow.dflow DEBUG: Task 2733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/43f8a1ab-10e2-41f5-b3a4-2f9e7b30876d_isrForCrosstalkSources_1125905_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/43f8a1ab-10e2-41f5-b3a4-2f9e7b30876d_isrForCrosstalkSources_1125905_36.stderr parsl.dataflow.dflow INFO: Task 2721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b6f2a1ada4a9a33f3d1c958b9d1a6eb with result from task 2721 parsl.dataflow.dflow DEBUG: Task 2721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/64e3e9bb-28c4-429e-9eeb-feb0dd5c21f8_isrForCrosstalkSources_1125929_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/64e3e9bb-28c4-429e-9eeb-feb0dd5c21f8_isrForCrosstalkSources_1125929_43.stderr parsl.dataflow.dflow INFO: Task 2811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dde4d59d4d80a953608a31ddc3ae619 with result from task 2811 parsl.dataflow.dflow DEBUG: Task 2811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2d5e6aab-2016-4a48-ae34-0e32951685b9_isrForCrosstalkSources_1125933_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2d5e6aab-2016-4a48-ae34-0e32951685b9_isrForCrosstalkSources_1125933_9.stderr parsl.dataflow.dflow INFO: Task 2751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a5e406d2ba111c0d93b1174726d2bb3 with result from task 2751 parsl.dataflow.dflow DEBUG: Task 2751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/38ff2949-383c-400a-816f-4b842240f710_isrForCrosstalkSources_1125903_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/38ff2949-383c-400a-816f-4b842240f710_isrForCrosstalkSources_1125903_10.stderr parsl.dataflow.dflow INFO: Task 2761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 138e09a7dcbc457932aef4b7e56ab6a6 with result from task 2761 parsl.dataflow.dflow DEBUG: Task 2761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/4ffa531b-b305-4d70-9edd-cb608dbeabf3_isrForCrosstalkSources_1125909_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/4ffa531b-b305-4d70-9edd-cb608dbeabf3_isrForCrosstalkSources_1125909_23.stderr parsl.dataflow.dflow INFO: Task 2789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a78723634ebeaf48b8af1f55dd23b4fe with result from task 2789 parsl.dataflow.dflow DEBUG: Task 2789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/0534139e-6f1f-4f51-a956-32a34b0e1d4b_isrForCrosstalkSources_1125944_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/0534139e-6f1f-4f51-a956-32a34b0e1d4b_isrForCrosstalkSources_1125944_18.stderr parsl.dataflow.dflow INFO: Task 2795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a6b9fba806c5aa47159b8c011e5ae2a with result from task 2795 parsl.dataflow.dflow DEBUG: Task 2795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/5f4dbea9-5458-4ed8-bb67-e18f66c0610d_isrForCrosstalkSources_1125924_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/5f4dbea9-5458-4ed8-bb67-e18f66c0610d_isrForCrosstalkSources_1125924_1.stderr parsl.dataflow.dflow INFO: Task 2687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76c6cc78ae44bc22d3b92fffb1536b82 with result from task 2687 parsl.dataflow.dflow DEBUG: Task 2687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/7659c1fc-c72f-4645-85ec-00f216602468_isrForCrosstalkSources_1125888_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/7659c1fc-c72f-4645-85ec-00f216602468_isrForCrosstalkSources_1125888_23.stderr parsl.dataflow.dflow INFO: Task 2725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52a2306ac361c687cae5c2de3746de51 with result from task 2725 parsl.dataflow.dflow DEBUG: Task 2725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/23709c35-78ae-431b-bd85-df6ca8177e29_isrForCrosstalkSources_1125924_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/23709c35-78ae-431b-bd85-df6ca8177e29_isrForCrosstalkSources_1125924_28.stderr parsl.dataflow.dflow INFO: Task 2805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4db839bac8b2ffe44f2d7166b0703fb3 with result from task 2805 parsl.dataflow.dflow DEBUG: Task 2805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/f8c7d772-8a2a-436a-8d8a-8ddcd09d3c71_isrForCrosstalkSources_1125934_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/f8c7d772-8a2a-436a-8d8a-8ddcd09d3c71_isrForCrosstalkSources_1125934_23.stderr parsl.dataflow.dflow INFO: Task 2746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6872c2238321e463a9ec4cb796a1a7a with result from task 2746 parsl.dataflow.dflow DEBUG: Task 2746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/7a0be123-b460-4f9c-a37c-dbda44f93a5e_isrForCrosstalkSources_1125933_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/7a0be123-b460-4f9c-a37c-dbda44f93a5e_isrForCrosstalkSources_1125933_1.stderr parsl.dataflow.dflow INFO: Task 2594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 346c582f101aedae9d9f7c993539c076 with result from task 2594 parsl.dataflow.dflow DEBUG: Task 2594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/8ff18580-922c-4e32-b6ec-b14e88ea640a_isrForCrosstalkSources_1125930_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/8ff18580-922c-4e32-b6ec-b14e88ea640a_isrForCrosstalkSources_1125930_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d228e699d7b8d4fbfb1ad3d09b123578 with result from task 2727 parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:05 proc_lsst.multi:146 [INFO] found job 16896459 in provider checkpoint proc_lsst.multi INFO: found job 16896459 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/8a8805f5-b152-454c-bc2d-3a4d7cfc8c8e_isrForCrosstalkSources_1125946_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/8a8805f5-b152-454c-bc2d-3a4d7cfc8c8e_isrForCrosstalkSources_1125946_22.stderr parsl.dataflow.dflow INFO: Task 2782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c79721768862789ebabeab1830392159 with result from task 2782 parsl.dataflow.dflow DEBUG: Task 2782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/762a2b15-3311-4fa7-848b-daca12129a6a_isrForCrosstalkSources_1125892_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/762a2b15-3311-4fa7-848b-daca12129a6a_isrForCrosstalkSources_1125892_19.stderr parsl.dataflow.dflow INFO: Task 2740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ee4f9fc445468dc38c32ce01b261b00 with result from task 2740 parsl.dataflow.dflow DEBUG: Task 2740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/124388d2-e7f6-4e73-a3a0-513f4198e3a6_isrForCrosstalkSources_1125908_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/124388d2-e7f6-4e73-a3a0-513f4198e3a6_isrForCrosstalkSources_1125908_48.stderr parsl.dataflow.dflow INFO: Task 2748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cb726e6f720901d86178fa22cae4686 with result from task 2748 parsl.dataflow.dflow DEBUG: Task 2748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/bbd3a262-90ae-4eab-8b3f-5be3f0a6a1de_isrForCrosstalkSources_1125917_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/bbd3a262-90ae-4eab-8b3f-5be3f0a6a1de_isrForCrosstalkSources_1125917_61.stderr parsl.dataflow.dflow INFO: Task 2787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2c8b3aff2f903f8afe0065b254ddc2f with result from task 2787 parsl.dataflow.dflow DEBUG: Task 2787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/f048a2f6-9270-42e2-b257-66af6e012086_isrForCrosstalkSources_1125891_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/f048a2f6-9270-42e2-b257-66af6e012086_isrForCrosstalkSources_1125891_61.stderr parsl.dataflow.dflow INFO: Task 2729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecff5967a11327ab2876826797d6a381 with result from task 2729 parsl.dataflow.dflow DEBUG: Task 2729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/f5f1ba00-35d5-441a-b48c-587118ce5607_isrForCrosstalkSources_1125946_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/f5f1ba00-35d5-441a-b48c-587118ce5607_isrForCrosstalkSources_1125946_40.stderr parsl.dataflow.dflow INFO: Task 2736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47184abb9aa5fac1e5dfc346e53900d7 with result from task 2736 parsl.dataflow.dflow DEBUG: Task 2736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/10d84562-ab83-434e-aa0c-3ca7a9f39cc0_isrForCrosstalkSources_1125937_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/10d84562-ab83-434e-aa0c-3ca7a9f39cc0_isrForCrosstalkSources_1125937_22.stderr parsl.dataflow.dflow INFO: Task 2739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be99958532dcb787d1f9f0f9f9f48b9c with result from task 2739 parsl.dataflow.dflow DEBUG: Task 2739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/1829332f-a6bc-422a-ae6d-15119a4185e8_isrForCrosstalkSources_1125899_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/1829332f-a6bc-422a-ae6d-15119a4185e8_isrForCrosstalkSources_1125899_56.stderr parsl.dataflow.dflow INFO: Task 2794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8d47b729e6560e7f36ab7dacb2678ea with result from task 2794 parsl.dataflow.dflow DEBUG: Task 2794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/a0ee050e-b132-4c67-a6d8-b6c2f6b07482_isrForCrosstalkSources_1125936_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/a0ee050e-b132-4c67-a6d8-b6c2f6b07482_isrForCrosstalkSources_1125936_6.stderr parsl.dataflow.dflow INFO: Task 2753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ba009d64c70bdab936d9c53b233d272 with result from task 2753 parsl.dataflow.dflow DEBUG: Task 2753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/977fcf50-ef21-46f1-b865-047cfd7b3fc2_isrForCrosstalkSources_1125933_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/977fcf50-ef21-46f1-b865-047cfd7b3fc2_isrForCrosstalkSources_1125933_35.stderr parsl.dataflow.dflow INFO: Task 2809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76d8768b39bed237d5ab0db44dd1371f with result from task 2809 parsl.dataflow.dflow DEBUG: Task 2809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/81fb1b82-8798-44ce-a588-50bca9de1173_isrForCrosstalkSources_1125902_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/81fb1b82-8798-44ce-a588-50bca9de1173_isrForCrosstalkSources_1125902_22.stderr parsl.dataflow.dflow INFO: Task 2712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd0bd10daf28550a01c03eb9090f1185 with result from task 2712 parsl.dataflow.dflow DEBUG: Task 2712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/71b578f8-e177-4c27-8c8e-697e9ce5b599_isrForCrosstalkSources_1125947_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/71b578f8-e177-4c27-8c8e-697e9ce5b599_isrForCrosstalkSources_1125947_55.stderr parsl.dataflow.dflow INFO: Task 2791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8888d3f7f64386b7c7acb0672af8d3c6 with result from task 2791 parsl.dataflow.dflow DEBUG: Task 2791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/b740d282-90d4-4a82-90bd-c38583c0439f_isrForCrosstalkSources_1125898_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/b740d282-90d4-4a82-90bd-c38583c0439f_isrForCrosstalkSources_1125898_10.stderr parsl.dataflow.dflow INFO: Task 2756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 007e43f28e5514a0a3dffe42a9523449 with result from task 2756 parsl.dataflow.dflow DEBUG: Task 2756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/4638d7f3-3ca2-4776-998b-9d2d79305bb8_isrForCrosstalkSources_1125946_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/4638d7f3-3ca2-4776-998b-9d2d79305bb8_isrForCrosstalkSources_1125946_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:06 proc_lsst.multi:146 [INFO] found job 16896460 in provider checkpoint proc_lsst.multi INFO: found job 16896460 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7a4cd55949585b4518f1e5506340221 with result from task 2776 parsl.dataflow.dflow DEBUG: Task 2776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/174f8687-8032-4102-ad15-61f4a71cafa1_isrForCrosstalkSources_1125947_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/174f8687-8032-4102-ad15-61f4a71cafa1_isrForCrosstalkSources_1125947_36.stderr parsl.dataflow.dflow INFO: Task 2757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfd551377e4c1d150fc215a1be569389 with result from task 2757 parsl.dataflow.dflow DEBUG: Task 2757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/cceec988-e459-428b-b223-a3b9d9d782ef_isrForCrosstalkSources_1125921_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/cceec988-e459-428b-b223-a3b9d9d782ef_isrForCrosstalkSources_1125921_14.stderr parsl.dataflow.dflow INFO: Task 2786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bc09cc2d0addb7dfdb5175d864cab72 with result from task 2786 parsl.dataflow.dflow DEBUG: Task 2786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/99c12d27-f111-4f54-9840-da29960abd08_isrForCrosstalkSources_1125906_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/99c12d27-f111-4f54-9840-da29960abd08_isrForCrosstalkSources_1125906_54.stderr parsl.dataflow.dflow INFO: Task 2766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f959375bfb8fbd2674cb37260525ae2d with result from task 2766 parsl.dataflow.dflow DEBUG: Task 2766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/39aa9f0b-8f4c-4b86-b118-41cf10d0fb51_isrForCrosstalkSources_1125924_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/39aa9f0b-8f4c-4b86-b118-41cf10d0fb51_isrForCrosstalkSources_1125924_44.stderr parsl.dataflow.dflow INFO: Task 2645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c58906601c92e4c293992d67add0bb0 with result from task 2645 parsl.dataflow.dflow DEBUG: Task 2645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/d2d8f177-b399-4da0-8cbb-6c62b590a90d_isrForCrosstalkSources_1125906_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/d2d8f177-b399-4da0-8cbb-6c62b590a90d_isrForCrosstalkSources_1125906_61.stderr parsl.dataflow.dflow INFO: Task 2747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9abc0249b7e84b88068063a7bdfc5294 with result from task 2747 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ccc1a43f-4bf6-4dc0-9002-92fd5f108931_isrForCrosstalkSources_1125939_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/ccc1a43f-4bf6-4dc0-9002-92fd5f108931_isrForCrosstalkSources_1125939_53.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:06 proc_lsst.multi:146 [INFO] found job 16896461 in provider checkpoint proc_lsst.multi INFO: found job 16896461 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 896541510877931d2fc88c7eece55d97 with result from task 2680 parsl.dataflow.dflow DEBUG: Task 2680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/44799503-75a1-4145-ace5-3440568aac63_isrForCrosstalkSources_1125926_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/44799503-75a1-4145-ace5-3440568aac63_isrForCrosstalkSources_1125926_31.stderr parsl.dataflow.dflow INFO: Task 2686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3980e6d60bc7382d1b507171bfa696da with result from task 2686 parsl.dataflow.dflow DEBUG: Task 2686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/6641d0a6-01a6-496f-a04e-0f6ee4b18b1a_isrForCrosstalkSources_1125906_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/6641d0a6-01a6-496f-a04e-0f6ee4b18b1a_isrForCrosstalkSources_1125906_15.stderr parsl.dataflow.dflow INFO: Task 2800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce98bd7a141a812a9aa4aa28d0cc0170 with result from task 2800 parsl.dataflow.dflow DEBUG: Task 2800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/ae06c30b-4bb9-45bf-a7d2-25d0b022c9d1_isrForCrosstalkSources_1125941_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/ae06c30b-4bb9-45bf-a7d2-25d0b022c9d1_isrForCrosstalkSources_1125941_46.stderr parsl.dataflow.dflow INFO: Task 2678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a93df66b5a6137e8aaa42f4b5086708d with result from task 2678 parsl.dataflow.dflow DEBUG: Task 2678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/7d4a7222-0301-48b2-b58f-fb2f85f810ea_isrForCrosstalkSources_1125920_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/7d4a7222-0301-48b2-b58f-fb2f85f810ea_isrForCrosstalkSources_1125920_38.stderr parsl.dataflow.dflow INFO: Task 2698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edc40e57502d448ac66b31f2ddb0d436 with result from task 2698 parsl.dataflow.dflow DEBUG: Task 2698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/57e32e71-6d86-4516-9d42-9755896c6e0e_isrForCrosstalkSources_1125932_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/57e32e71-6d86-4516-9d42-9755896c6e0e_isrForCrosstalkSources_1125932_50.stderr parsl.dataflow.dflow INFO: Task 2834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ad64ad187d4f6b8dddba00ed200d289 with result from task 2834 parsl.dataflow.dflow DEBUG: Task 2834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/9f6dbb8d-7312-43e3-82b0-f0c4f3b118ef_isrForCrosstalkSources_1125891_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/9f6dbb8d-7312-43e3-82b0-f0c4f3b118ef_isrForCrosstalkSources_1125891_32.stderr parsl.dataflow.dflow INFO: Task 2738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bd5dce3dd2492d589829ae132db6b06 with result from task 2738 parsl.dataflow.dflow DEBUG: Task 2738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/97232421-cab8-4125-93bd-277c158c4dee_isrForCrosstalkSources_1125897_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/97232421-cab8-4125-93bd-277c158c4dee_isrForCrosstalkSources_1125897_9.stderr parsl.dataflow.dflow INFO: Task 2835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b0ee2818dadc5c3316aa626f0e79be8 with result from task 2835 parsl.dataflow.dflow DEBUG: Task 2835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/0dbc4433-0850-45c2-badb-0969c91b0a18_isrForCrosstalkSources_1125898_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/0dbc4433-0850-45c2-badb-0969c91b0a18_isrForCrosstalkSources_1125898_21.stderr parsl.dataflow.dflow INFO: Task 2749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 111dd858d56bb005a215cf4876a2a828 with result from task 2749 parsl.dataflow.dflow DEBUG: Task 2749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/4749fec9-29e1-4e8e-863b-e9be2bd9c2e4_isrForCrosstalkSources_1125910_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/4749fec9-29e1-4e8e-863b-e9be2bd9c2e4_isrForCrosstalkSources_1125910_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:07 proc_lsst.multi:146 [INFO] found job 16896462 in provider checkpoint proc_lsst.multi INFO: found job 16896462 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40979990b2f483db7e80dda946b8e645 with result from task 2737 parsl.dataflow.dflow DEBUG: Task 2737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/889f91c1-39b8-433c-b62e-8136e76bbd37_isrForCrosstalkSources_1125888_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/889f91c1-39b8-433c-b62e-8136e76bbd37_isrForCrosstalkSources_1125888_58.stderr parsl.dataflow.dflow INFO: Task 2831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39241c9e1965f521759a59f83dc298df with result from task 2831 parsl.dataflow.dflow DEBUG: Task 2831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/03bdb4ce-17b2-46d5-aaf7-93fb9e86c180_isrForCrosstalkSources_1125896_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/03bdb4ce-17b2-46d5-aaf7-93fb9e86c180_isrForCrosstalkSources_1125896_8.stderr parsl.dataflow.dflow INFO: Task 2815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a3608fd74fdf24cbda83379ee5efb36 with result from task 2815 parsl.dataflow.dflow DEBUG: Task 2815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/23dc16b4-92a2-4953-9dff-1a767f2b4d6a_isrForCrosstalkSources_1125941_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/23dc16b4-92a2-4953-9dff-1a767f2b4d6a_isrForCrosstalkSources_1125941_24.stderr parsl.dataflow.dflow INFO: Task 2769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 450f30285af72286be22f0dfc3235612 with result from task 2769 parsl.dataflow.dflow DEBUG: Task 2769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/91398b70-7e05-41e1-8709-8fef969e286f_isrForCrosstalkSources_1125936_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/91398b70-7e05-41e1-8709-8fef969e286f_isrForCrosstalkSources_1125936_22.stderr parsl.dataflow.dflow INFO: Task 2807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9929f6f5d7c9e69cc78c2f3a83eccde with result from task 2807 parsl.dataflow.dflow DEBUG: Task 2807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/1cc5071b-86ec-4f8d-be84-9545622fdb8f_isrForCrosstalkSources_1125910_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/1cc5071b-86ec-4f8d-be84-9545622fdb8f_isrForCrosstalkSources_1125910_29.stderr parsl.dataflow.dflow INFO: Task 2849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbc9526405b1deaeea52f02ae1730873 with result from task 2849 parsl.dataflow.dflow DEBUG: Task 2849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/043832e2-1119-4ca4-b880-1c23e6bd9324_isrForCrosstalkSources_1125914_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/043832e2-1119-4ca4-b880-1c23e6bd9324_isrForCrosstalkSources_1125914_19.stderr parsl.dataflow.dflow INFO: Task 2781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6add338e3f8d3ec2ff0582f77734eb3c with result from task 2781 parsl.dataflow.dflow DEBUG: Task 2781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/533af3e2-a175-456f-b727-4764d0f1cc30_isrForCrosstalkSources_1125920_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/533af3e2-a175-456f-b727-4764d0f1cc30_isrForCrosstalkSources_1125920_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:07 proc_lsst.multi:146 [INFO] found job 16896463 in provider checkpoint proc_lsst.multi INFO: found job 16896463 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45a50f8cc6f972b98dafff4c3b58bad8 with result from task 2843 parsl.dataflow.dflow DEBUG: Task 2843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c8d1d42f-d6eb-4bc1-946a-f888fc0e1a00_isrForCrosstalkSources_1125910_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c8d1d42f-d6eb-4bc1-946a-f888fc0e1a00_isrForCrosstalkSources_1125910_51.stderr parsl.dataflow.dflow INFO: Task 2861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79f46d6c8858c2d98ec4a5e2b76b07db with result from task 2861 parsl.dataflow.dflow DEBUG: Task 2861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/22960df5-fd80-4a2c-9968-f9a1cda3a6ff_isrForCrosstalkSources_1125910_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/22960df5-fd80-4a2c-9968-f9a1cda3a6ff_isrForCrosstalkSources_1125910_21.stderr parsl.dataflow.dflow INFO: Task 2853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf46bd187b04d17fdbc8f00024233562 with result from task 2853 parsl.dataflow.dflow DEBUG: Task 2853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/a93bb9d6-7b00-481a-9ea6-b6b37b2dd753_isrForCrosstalkSources_1125943_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/a93bb9d6-7b00-481a-9ea6-b6b37b2dd753_isrForCrosstalkSources_1125943_40.stderr parsl.dataflow.dflow INFO: Task 2752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21afee80b277fe1b17eaf0a11cbad5f2 with result from task 2752 parsl.dataflow.dflow DEBUG: Task 2752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/6f4a180d-6208-48b5-aa1c-b4ec22a6d1ab_isrForCrosstalkSources_1125942_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/6f4a180d-6208-48b5-aa1c-b4ec22a6d1ab_isrForCrosstalkSources_1125942_42.stderr parsl.dataflow.dflow INFO: Task 2688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a0cf955c8c70499fa8227f815551982 with result from task 2688 parsl.dataflow.dflow DEBUG: Task 2688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9a67d732-6a7e-4e5e-bbae-2af4ed160398_isrForCrosstalkSources_1125934_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9a67d732-6a7e-4e5e-bbae-2af4ed160398_isrForCrosstalkSources_1125934_18.stderr parsl.dataflow.dflow INFO: Task 2819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3cb04560980b516d06b4ebd255490e6 with result from task 2819 parsl.dataflow.dflow DEBUG: Task 2819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/be82cb31-c497-4678-82aa-0b663c29441b_isrForCrosstalkSources_1125904_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/be82cb31-c497-4678-82aa-0b663c29441b_isrForCrosstalkSources_1125904_40.stderr parsl.dataflow.dflow INFO: Task 2825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c69529add055237012fe202e6597d149 with result from task 2825 parsl.dataflow.dflow DEBUG: Task 2825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/5b82f52d-fd6e-4a93-be8a-317a9fed036b_isrForCrosstalkSources_1125918_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/5b82f52d-fd6e-4a93-be8a-317a9fed036b_isrForCrosstalkSources_1125918_7.stderr parsl.dataflow.dflow INFO: Task 2774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 343bb129276a97f10106c12920c05be6 with result from task 2774 parsl.dataflow.dflow DEBUG: Task 2774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/df3003e6-fc0d-4d72-8c7f-97c3c93a8be9_isrForCrosstalkSources_1125914_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/df3003e6-fc0d-4d72-8c7f-97c3c93a8be9_isrForCrosstalkSources_1125914_3.stderr parsl.dataflow.dflow INFO: Task 2860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca3c63963154fc513999d5ab58f17c54 with result from task 2860 parsl.dataflow.dflow DEBUG: Task 2860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/a0cd4dce-eb29-4368-9ec7-a289ccd13cff_isrForCrosstalkSources_1125899_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/a0cd4dce-eb29-4368-9ec7-a289ccd13cff_isrForCrosstalkSources_1125899_3.stderr parsl.dataflow.dflow INFO: Task 2694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e207c50ab9854e7dafdc861afd36282 with result from task 2694 parsl.dataflow.dflow DEBUG: Task 2694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/297ddcad-9888-43b8-8ce6-c068861f18f6_isrForCrosstalkSources_1125943_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/297ddcad-9888-43b8-8ce6-c068861f18f6_isrForCrosstalkSources_1125943_7.stderr parsl.dataflow.dflow INFO: Task 2743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58aa33463ac2fed56074eefd26db9a78 with result from task 2743 parsl.dataflow.dflow DEBUG: Task 2743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/ba5d58db-4484-4cb4-9ad2-988302f7d181_isrForCrosstalkSources_1125929_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/ba5d58db-4484-4cb4-9ad2-988302f7d181_isrForCrosstalkSources_1125929_57.stderr parsl.dataflow.dflow INFO: Task 2841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09acfa7c3f5527521790fe3ad836c84d with result from task 2841 parsl.dataflow.dflow DEBUG: Task 2841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/f716563d-37de-413e-b5a7-51b549d28e0b_isrForCrosstalkSources_1125903_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/f716563d-37de-413e-b5a7-51b549d28e0b_isrForCrosstalkSources_1125903_11.stderr parsl.dataflow.dflow INFO: Task 2799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad8f586bb4be709dd521cb63df9be4eb with result from task 2799 parsl.dataflow.dflow DEBUG: Task 2799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4797a256-d69e-426b-895c-edfd8170a487_isrForCrosstalkSources_1125893_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4797a256-d69e-426b-895c-edfd8170a487_isrForCrosstalkSources_1125893_30.stderr parsl.dataflow.dflow INFO: Task 2822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 478484f0eb762ae493e59fec0b2cfd22 with result from task 2822 parsl.dataflow.dflow DEBUG: Task 2822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/880e3b6a-446c-49fa-a8f5-6aa62c596821_isrForCrosstalkSources_1125905_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/880e3b6a-446c-49fa-a8f5-6aa62c596821_isrForCrosstalkSources_1125905_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:08 proc_lsst.multi:146 [INFO] found job 16896464 in provider checkpoint proc_lsst.multi INFO: found job 16896464 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 115c8259c78d1a52dd9b9fac2b1e20e8 with result from task 2826 parsl.dataflow.dflow DEBUG: Task 2826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/4d33fd38-6ea5-4825-95c0-625a61dd56e2_isrForCrosstalkSources_1125918_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/4d33fd38-6ea5-4825-95c0-625a61dd56e2_isrForCrosstalkSources_1125918_2.stderr parsl.dataflow.dflow INFO: Task 2827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c11023c67dfee9909f00596293cd9bec with result from task 2827 parsl.dataflow.dflow DEBUG: Task 2827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/82649a1e-c218-41ae-a121-6e9fcced4a8f_isrForCrosstalkSources_1125947_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/82649a1e-c218-41ae-a121-6e9fcced4a8f_isrForCrosstalkSources_1125947_15.stderr parsl.dataflow.dflow INFO: Task 2833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3735deec2425ecd8090baf8ce4f1c9ae with result from task 2833 parsl.dataflow.dflow DEBUG: Task 2833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/abc71bb4-e246-40f4-8c1d-6741f98d1061_isrForCrosstalkSources_1125909_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/abc71bb4-e246-40f4-8c1d-6741f98d1061_isrForCrosstalkSources_1125909_8.stderr parsl.dataflow.dflow INFO: Task 2801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cdd435694fa39a0a4b9dc24e21b739a with result from task 2801 parsl.dataflow.dflow DEBUG: Task 2801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/fe3d892e-80d6-472c-9438-53cc9ad93e55_isrForCrosstalkSources_1125901_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/fe3d892e-80d6-472c-9438-53cc9ad93e55_isrForCrosstalkSources_1125901_42.stderr parsl.dataflow.dflow INFO: Task 2836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 603122e6db89fecfb60e9b21df128270 with result from task 2836 parsl.dataflow.dflow DEBUG: Task 2836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/c48c1af7-dc2b-40cc-a272-5f56194c119e_isrForCrosstalkSources_1125920_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/c48c1af7-dc2b-40cc-a272-5f56194c119e_isrForCrosstalkSources_1125920_60.stderr parsl.dataflow.dflow INFO: Task 2829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01897d556e6fe6664dccc31f109beeb1 with result from task 2829 parsl.dataflow.dflow DEBUG: Task 2829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/c10b9fb5-c528-4ce7-b86f-3c267abe70b8_isrForCrosstalkSources_1125888_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/c10b9fb5-c528-4ce7-b86f-3c267abe70b8_isrForCrosstalkSources_1125888_49.stderr parsl.dataflow.dflow INFO: Task 2838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcd9416dc2c470fe3ccdd79817d28239 with result from task 2838 parsl.dataflow.dflow DEBUG: Task 2838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/80a79cf1-37c5-4ccf-9e9f-8d831ccbd199_isrForCrosstalkSources_1125933_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/80a79cf1-37c5-4ccf-9e9f-8d831ccbd199_isrForCrosstalkSources_1125933_58.stderr parsl.dataflow.dflow INFO: Task 2804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38f9fdd1f42942f3d4d2eac60a45ea2d with result from task 2804 parsl.dataflow.dflow DEBUG: Task 2804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/798eaa2d-6545-4d53-ae12-8d8b502f6d48_isrForCrosstalkSources_1125943_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/798eaa2d-6545-4d53-ae12-8d8b502f6d48_isrForCrosstalkSources_1125943_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:08 proc_lsst.multi:146 [INFO] found job 16896466 in provider checkpoint proc_lsst.multi INFO: found job 16896466 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef281ac1ab180f5d01c95d83afedf229 with result from task 2806 parsl.dataflow.dflow DEBUG: Task 2806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/c007615b-df2f-452f-8bdd-c526d9f0ae59_isrForCrosstalkSources_1125907_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/c007615b-df2f-452f-8bdd-c526d9f0ae59_isrForCrosstalkSources_1125907_30.stderr parsl.dataflow.dflow INFO: Task 2854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 986a7f6fbc3b80ab37bafd330748319b with result from task 2854 parsl.dataflow.dflow DEBUG: Task 2854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/bd4d8fc4-8c14-4114-86fe-e6ef16f88f85_isrForCrosstalkSources_1125888_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/bd4d8fc4-8c14-4114-86fe-e6ef16f88f85_isrForCrosstalkSources_1125888_25.stderr parsl.dataflow.dflow INFO: Task 2697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8c8551d8cfbe00c31b7576342c4a005 with result from task 2697 parsl.dataflow.dflow DEBUG: Task 2697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/dec4254c-f626-426f-95db-07cc29f9c31f_isrForCrosstalkSources_1125911_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/dec4254c-f626-426f-95db-07cc29f9c31f_isrForCrosstalkSources_1125911_60.stderr parsl.dataflow.dflow INFO: Task 2850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ac6502095b08310b21eb4c3adb8b10b with result from task 2850 parsl.dataflow.dflow DEBUG: Task 2850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/16a21452-f209-46e1-b3ca-ef7d27354137_isrForCrosstalkSources_1125940_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/16a21452-f209-46e1-b3ca-ef7d27354137_isrForCrosstalkSources_1125940_5.stderr parsl.dataflow.dflow INFO: Task 2874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1677bdc651611a4388b79e9ce31488b0 with result from task 2874 parsl.dataflow.dflow DEBUG: Task 2874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/ace705b0-afba-420e-9f16-0157e47ec7e2_isrForCrosstalkSources_1125925_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/ace705b0-afba-420e-9f16-0157e47ec7e2_isrForCrosstalkSources_1125925_5.stderr parsl.dataflow.dflow INFO: Task 2704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 231d417a68fe8106e976d3d3705e008f with result from task 2704 parsl.dataflow.dflow DEBUG: Task 2704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/842b1e47-2ffa-442e-a19f-dccbaae95548_isrForCrosstalkSources_1125933_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/842b1e47-2ffa-442e-a19f-dccbaae95548_isrForCrosstalkSources_1125933_47.stderr parsl.dataflow.dflow INFO: Task 2828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7876fb9b08965afcec2b1a0cad1ddb07 with result from task 2828 parsl.dataflow.dflow DEBUG: Task 2828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/963c37a9-fa0e-4bbd-8515-bc043ac4da14_isrForCrosstalkSources_1125920_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/963c37a9-fa0e-4bbd-8515-bc043ac4da14_isrForCrosstalkSources_1125920_9.stderr parsl.dataflow.dflow INFO: Task 2866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d95a0f4c383d3cb4c415809889828a6 with result from task 2866 parsl.dataflow.dflow DEBUG: Task 2866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/f91ad774-67fe-4513-9cd6-8e8d5ca9af9b_isrForCrosstalkSources_1125896_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/f91ad774-67fe-4513-9cd6-8e8d5ca9af9b_isrForCrosstalkSources_1125896_23.stderr parsl.dataflow.dflow INFO: Task 2709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a82993e92a2e341f33353d80d3e35ff1 with result from task 2709 parsl.dataflow.dflow DEBUG: Task 2709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/df3d789d-4746-4ac9-b7e5-35878a20c064_isrForCrosstalkSources_1125912_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/df3d789d-4746-4ac9-b7e5-35878a20c064_isrForCrosstalkSources_1125912_30.stderr parsl.dataflow.dflow INFO: Task 2689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7318583362a6ff6a1d0a155029191ae2 with result from task 2689 parsl.dataflow.dflow DEBUG: Task 2689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/04c6bb87-387d-4807-8483-0349ccec9678_isrForCrosstalkSources_1125915_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/04c6bb87-387d-4807-8483-0349ccec9678_isrForCrosstalkSources_1125915_30.stderr parsl.dataflow.dflow INFO: Task 2830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9bd19d54850d473736d865b71619f27 with result from task 2830 parsl.dataflow.dflow DEBUG: Task 2830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/ad2de96f-5128-4ab6-a372-73af13bb824e_isrForCrosstalkSources_1125900_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/ad2de96f-5128-4ab6-a372-73af13bb824e_isrForCrosstalkSources_1125900_19.stderr parsl.dataflow.dflow INFO: Task 2679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry def429cb13d3882680276d2a35716b4b with result from task 2679 parsl.dataflow.dflow DEBUG: Task 2679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/91a59eba-d448-4e67-86db-e9478a94c340_isrForCrosstalkSources_1125893_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/91a59eba-d448-4e67-86db-e9478a94c340_isrForCrosstalkSources_1125893_49.stderr parsl.dataflow.dflow INFO: Task 2875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db4a95ac85ced6fa30197d6a7ceb3901 with result from task 2875 parsl.dataflow.dflow DEBUG: Task 2875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/5923154c-2c1e-4748-9378-e1e1ca317e2a_isrForCrosstalkSources_1125896_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/5923154c-2c1e-4748-9378-e1e1ca317e2a_isrForCrosstalkSources_1125896_36.stderr parsl.dataflow.dflow INFO: Task 2667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f7b9c14e9bc7dc16194dbf03410e38d with result from task 2667 parsl.dataflow.dflow DEBUG: Task 2667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/5aa2fb4d-e3a1-4c23-bf6b-1bfa2ddf9553_isrForCrosstalkSources_1125938_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/5aa2fb4d-e3a1-4c23-bf6b-1bfa2ddf9553_isrForCrosstalkSources_1125938_27.stderr parsl.dataflow.dflow INFO: Task 2808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6de9a69ec87da987bdcafcb4b7576b6 with result from task 2808 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/42c770b0-36a6-45f9-ae54-bc3b51fb4824_isrForCrosstalkSources_1125922_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/42c770b0-36a6-45f9-ae54-bc3b51fb4824_isrForCrosstalkSources_1125922_53.stderr parsl.dataflow.dflow INFO: Task 2797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cb2e9e4b842b687a12e2113af508f6e with result from task 2797 parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:09 proc_lsst.multi:146 [INFO] found job 16896467 in provider checkpoint proc_lsst.multi INFO: found job 16896467 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/4db6714c-7c98-4d37-ade7-2e169c680e1d_isrForCrosstalkSources_1125917_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/4db6714c-7c98-4d37-ade7-2e169c680e1d_isrForCrosstalkSources_1125917_57.stderr parsl.dataflow.dflow INFO: Task 2796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 713322984a9b1a61e24f41ed44fc17a2 with result from task 2796 parsl.dataflow.dflow DEBUG: Task 2796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/546543ac-e95c-43b5-8c63-83e83edbfa2b_isrForCrosstalkSources_1125935_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/546543ac-e95c-43b5-8c63-83e83edbfa2b_isrForCrosstalkSources_1125935_13.stderr parsl.dataflow.dflow INFO: Task 2764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd1b3310057a34187bde5c14e130ed7b with result from task 2764 parsl.dataflow.dflow DEBUG: Task 2764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/b69a0e1b-bfa6-40d6-a484-820457e030b0_isrForCrosstalkSources_1125907_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/b69a0e1b-bfa6-40d6-a484-820457e030b0_isrForCrosstalkSources_1125907_21.stderr parsl.dataflow.dflow INFO: Task 2863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 016d32f7e418d25db65eb66e8f9cd9df with result from task 2863 parsl.dataflow.dflow DEBUG: Task 2863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/a7feb9d5-d6cf-4249-b2b2-8084ce9c85fc_isrForCrosstalkSources_1125922_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/a7feb9d5-d6cf-4249-b2b2-8084ce9c85fc_isrForCrosstalkSources_1125922_40.stderr parsl.dataflow.dflow INFO: Task 2857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 446dd8d5e013ec39eb632f03d18e6e8e with result from task 2857 parsl.dataflow.dflow DEBUG: Task 2857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/00a5bf29-afcf-4ee4-b36a-26d57c67a142_isrForCrosstalkSources_1125940_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/00a5bf29-afcf-4ee4-b36a-26d57c67a142_isrForCrosstalkSources_1125940_8.stderr parsl.dataflow.dflow INFO: Task 2846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 514517acf9de4ca3c8af325de9980b46 with result from task 2846 parsl.dataflow.dflow DEBUG: Task 2846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c0ab949b-cdfe-4f18-a69c-25181db83f50_isrForCrosstalkSources_1125893_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c0ab949b-cdfe-4f18-a69c-25181db83f50_isrForCrosstalkSources_1125893_57.stderr parsl.dataflow.dflow INFO: Task 2775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea80c34d7da0e57830acff68630a9602 with result from task 2775 parsl.dataflow.dflow DEBUG: Task 2775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/45b0ba7e-b9e8-432a-8667-e46b2e859e55_isrForCrosstalkSources_1125923_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/45b0ba7e-b9e8-432a-8667-e46b2e859e55_isrForCrosstalkSources_1125923_53.stderr parsl.dataflow.dflow INFO: Task 2864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76d1a3265ca5acc95b9f86d0edd1056d with result from task 2864 parsl.dataflow.dflow DEBUG: Task 2864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/815304be-8498-4739-bd5f-4af221b21e86_isrForCrosstalkSources_1125908_40.stdout parsl.dataflow.dflow INFO: Standard error for task 2864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/815304be-8498-4739-bd5f-4af221b21e86_isrForCrosstalkSources_1125908_40.stderr parsl.dataflow.dflow INFO: Task 2856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bf6a868c07a7d1c72882e9059fa35e2 with result from task 2856 parsl.dataflow.dflow DEBUG: Task 2856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/4db929b6-cb68-45e3-abc7-bda5ed82892f_isrForCrosstalkSources_1125945_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/4db929b6-cb68-45e3-abc7-bda5ed82892f_isrForCrosstalkSources_1125945_45.stderr parsl.dataflow.dflow INFO: Task 2873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1320fda605fea71a1b986f79c500ffb with result from task 2873 parsl.dataflow.dflow DEBUG: Task 2873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/269bc177-032e-4bf1-921c-3974acae78ee_isrForCrosstalkSources_1125903_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/269bc177-032e-4bf1-921c-3974acae78ee_isrForCrosstalkSources_1125903_21.stderr parsl.dataflow.dflow INFO: Task 2840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17bce79795571fdecf304486c847dc9c with result from task 2840 parsl.dataflow.dflow DEBUG: Task 2840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/b05112a2-3381-4640-8842-5adf5805c608_isrForCrosstalkSources_1125919_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/b05112a2-3381-4640-8842-5adf5805c608_isrForCrosstalkSources_1125919_56.stderr parsl.dataflow.dflow INFO: Task 2832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dc0bc5009c35e3e55c70050bb48536a with result from task 2832 parsl.dataflow.dflow DEBUG: Task 2832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d3a2674a-2ab6-40f6-b356-5600ff647046_isrForCrosstalkSources_1125911_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d3a2674a-2ab6-40f6-b356-5600ff647046_isrForCrosstalkSources_1125911_9.stderr parsl.dataflow.dflow INFO: Task 2651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa35f58653d92aba3825bebab0b7ee94 with result from task 2651 parsl.dataflow.dflow DEBUG: Task 2651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/70d48dac-c112-429d-bdc1-b19e56ed4150_isrForCrosstalkSources_1125889_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/70d48dac-c112-429d-bdc1-b19e56ed4150_isrForCrosstalkSources_1125889_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:09 proc_lsst.multi:146 [INFO] found job 16896468 in provider checkpoint proc_lsst.multi INFO: found job 16896468 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9259b0f6e61ec8a125652210ac417c19 with result from task 2760 parsl.dataflow.dflow DEBUG: Task 2760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/b996cee8-214d-412c-b52b-08b6f2504df1_isrForCrosstalkSources_1125902_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/b996cee8-214d-412c-b52b-08b6f2504df1_isrForCrosstalkSources_1125902_61.stderr parsl.dataflow.dflow INFO: Task 2837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa521700e1711b0f137a1226fab306d9 with result from task 2837 parsl.dataflow.dflow DEBUG: Task 2837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/aa6a5575-1a4e-4ad5-af48-c9b53373215c_isrForCrosstalkSources_1125930_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/aa6a5575-1a4e-4ad5-af48-c9b53373215c_isrForCrosstalkSources_1125930_1.stderr parsl.dataflow.dflow INFO: Task 2888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bdda417e351d1d6fddb5375f743463c with result from task 2888 parsl.dataflow.dflow DEBUG: Task 2888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/4a3b9d5d-c750-4aa6-93b0-da1777c8a35b_isrForCrosstalkSources_1125928_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/4a3b9d5d-c750-4aa6-93b0-da1777c8a35b_isrForCrosstalkSources_1125928_16.stderr parsl.dataflow.dflow INFO: Task 2677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd844ad159431cc8945951cd7571deac with result from task 2677 parsl.dataflow.dflow DEBUG: Task 2677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6ad3345d-87bc-4a96-b8e0-fe8c7b1701cd_isrForCrosstalkSources_1125927_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6ad3345d-87bc-4a96-b8e0-fe8c7b1701cd_isrForCrosstalkSources_1125927_34.stderr parsl.dataflow.dflow INFO: Task 2858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69e36ca324643f4438ed80095406dba6 with result from task 2858 parsl.dataflow.dflow DEBUG: Task 2858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/fc221f2b-3c3a-4fda-9486-999157483956_isrForCrosstalkSources_1125901_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/fc221f2b-3c3a-4fda-9486-999157483956_isrForCrosstalkSources_1125901_36.stderr parsl.dataflow.dflow INFO: Task 2882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dacd8ed5668897232b8099ad3b8a8c4 with result from task 2882 parsl.dataflow.dflow DEBUG: Task 2882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/06a1204a-9bfe-46e1-90d7-555e6300e18b_isrForCrosstalkSources_1125899_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/06a1204a-9bfe-46e1-90d7-555e6300e18b_isrForCrosstalkSources_1125899_31.stderr parsl.dataflow.dflow INFO: Task 2855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ae36d45b66f6c8f4973838581bee80a with result from task 2855 parsl.dataflow.dflow DEBUG: Task 2855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1e5af2d0-573a-4428-98d3-d872ca8cb70d_isrForCrosstalkSources_1125900_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1e5af2d0-573a-4428-98d3-d872ca8cb70d_isrForCrosstalkSources_1125900_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:10 proc_lsst.multi:146 [INFO] found job 16896469 in provider checkpoint proc_lsst.multi INFO: found job 16896469 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bf9a0da6a60b4df4fd058847a6c6b8e with result from task 2851 parsl.dataflow.dflow DEBUG: Task 2851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/640b5a39-2772-49b8-8de3-dfe4121c8a6e_isrForCrosstalkSources_1125899_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/640b5a39-2772-49b8-8de3-dfe4121c8a6e_isrForCrosstalkSources_1125899_30.stderr parsl.dataflow.dflow INFO: Task 2859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b894db83751f9c486758b32a44d817a with result from task 2859 parsl.dataflow.dflow DEBUG: Task 2859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/1a5a4af6-1538-4e07-9d6c-b1dcc0fa6827_isrForCrosstalkSources_1125913_30.stdout parsl.dataflow.dflow INFO: Standard error for task 2859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/1a5a4af6-1538-4e07-9d6c-b1dcc0fa6827_isrForCrosstalkSources_1125913_30.stderr parsl.dataflow.dflow INFO: Task 2891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 982af632f66800c5280151042ee5877b with result from task 2891 parsl.dataflow.dflow DEBUG: Task 2891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/884c5c08-a1f1-42a3-b111-fbda77c0e36a_isrForCrosstalkSources_1125902_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/884c5c08-a1f1-42a3-b111-fbda77c0e36a_isrForCrosstalkSources_1125902_54.stderr parsl.dataflow.dflow INFO: Task 2876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c941bfba016de19251be852d47706215 with result from task 2876 parsl.dataflow.dflow DEBUG: Task 2876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d6210e4b-cfda-4975-91ad-e372681d0ca8_isrForCrosstalkSources_1125890_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/d6210e4b-cfda-4975-91ad-e372681d0ca8_isrForCrosstalkSources_1125890_58.stderr parsl.dataflow.dflow INFO: Task 2812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67d35bdab223a01d9d111649ba830a04 with result from task 2812 parsl.dataflow.dflow DEBUG: Task 2812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/49f13f50-3ea7-4414-83c5-b282467fd757_isrForCrosstalkSources_1125896_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/49f13f50-3ea7-4414-83c5-b282467fd757_isrForCrosstalkSources_1125896_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:10 proc_lsst.multi:146 [INFO] found job 16896470 in provider checkpoint proc_lsst.multi INFO: found job 16896470 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09030799e1dbdd2949c79f3ec8676ba8 with result from task 2839 parsl.dataflow.dflow DEBUG: Task 2839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/0eca967e-8fd0-45ae-bfd9-771e6f06aab5_isrForCrosstalkSources_1125924_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/0eca967e-8fd0-45ae-bfd9-771e6f06aab5_isrForCrosstalkSources_1125924_52.stderr parsl.dataflow.dflow INFO: Task 2885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4624a739e6d8ed01e6210b6c6edba95 with result from task 2885 parsl.dataflow.dflow DEBUG: Task 2885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/8a106361-b393-42a0-8072-f9f96de6c379_isrForCrosstalkSources_1125941_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/8a106361-b393-42a0-8072-f9f96de6c379_isrForCrosstalkSources_1125941_10.stderr parsl.dataflow.dflow INFO: Task 2821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c84b9a2c44fb88330ba43bba5d3cadf9 with result from task 2821 parsl.dataflow.dflow DEBUG: Task 2821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/2b4ce0fe-e41e-4ac9-8269-57e21af1a1c1_isrForCrosstalkSources_1125905_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/2b4ce0fe-e41e-4ac9-8269-57e21af1a1c1_isrForCrosstalkSources_1125905_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:11 proc_lsst.multi:146 [INFO] found job 16896471 in provider checkpoint proc_lsst.multi INFO: found job 16896471 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6607bc192a44a345af9cb1603cbd419f with result from task 2783 parsl.dataflow.dflow DEBUG: Task 2783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/7c948243-ff04-491b-af04-82505e444ed3_isrForCrosstalkSources_1125907_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/7c948243-ff04-491b-af04-82505e444ed3_isrForCrosstalkSources_1125907_38.stderr parsl.dataflow.dflow INFO: Task 2900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fa0654649b0f6e5aae9b3fc3a333e51 with result from task 2900 parsl.dataflow.dflow DEBUG: Task 2900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a6bbeb51-b0fa-4755-8ec1-97c618bdbdea_isrForCrosstalkSources_1125909_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a6bbeb51-b0fa-4755-8ec1-97c618bdbdea_isrForCrosstalkSources_1125909_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:11 proc_lsst.multi:146 [INFO] found job 16896472 in provider checkpoint proc_lsst.multi INFO: found job 16896472 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a76b99e0d503b920b9d38090c01c75c with result from task 2622 parsl.dataflow.dflow DEBUG: Task 2622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/ed489381-93de-427c-a576-aae78c4381f0_isrForCrosstalkSources_1125925_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/ed489381-93de-427c-a576-aae78c4381f0_isrForCrosstalkSources_1125925_39.stderr parsl.dataflow.dflow INFO: Task 2868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9f508c1b351d20cba651c2349798654 with result from task 2868 parsl.dataflow.dflow DEBUG: Task 2868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/fe5881cc-a96e-42f4-a662-3c287a1c6096_isrForCrosstalkSources_1125926_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/fe5881cc-a96e-42f4-a662-3c287a1c6096_isrForCrosstalkSources_1125926_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:11 proc_lsst.multi:146 [INFO] found job 16896473 in provider checkpoint proc_lsst.multi INFO: found job 16896473 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2071a072e1c6435fbb9ff5d9930705a3 with result from task 2790 parsl.dataflow.dflow DEBUG: Task 2790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/1a218824-5745-4d75-ae14-c29312500c29_isrForCrosstalkSources_1125935_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/1a218824-5745-4d75-ae14-c29312500c29_isrForCrosstalkSources_1125935_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:11 proc_lsst.multi:146 [INFO] found job 16896474 in provider checkpoint proc_lsst.multi INFO: found job 16896474 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14e2b808e05dfac0b7e1353a1b1d3506 with result from task 2913 parsl.dataflow.dflow DEBUG: Task 2913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/06bb6c91-81a0-46da-b448-8c0c4cb5a1fd_isrForCrosstalkSources_1125934_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/06bb6c91-81a0-46da-b448-8c0c4cb5a1fd_isrForCrosstalkSources_1125934_58.stderr parsl.dataflow.dflow INFO: Task 2779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 379246d98f10bee71aa0d866aeb79513 with result from task 2779 parsl.dataflow.dflow DEBUG: Task 2779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/ca4d024e-2c8d-4b36-94cd-cdb21cc15dd7_isrForCrosstalkSources_1125926_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/ca4d024e-2c8d-4b36-94cd-cdb21cc15dd7_isrForCrosstalkSources_1125926_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:11 proc_lsst.multi:146 [INFO] found job 16896475 in provider checkpoint proc_lsst.multi INFO: found job 16896475 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f631a4600b066fa710b51ddfe859eac3 with result from task 2953 parsl.dataflow.dflow DEBUG: Task 2953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/6e453841-3110-48b6-b8a9-ea608251336a_isrForCrosstalkSources_1125911_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/6e453841-3110-48b6-b8a9-ea608251336a_isrForCrosstalkSources_1125911_54.stderr parsl.dataflow.dflow INFO: Task 2908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7f6005bc9473d0ac3c4001fc66856f8 with result from task 2908 parsl.dataflow.dflow DEBUG: Task 2908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/376f8d69-b7c3-443d-9f54-4185d44bb6cf_isrForCrosstalkSources_1125917_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/376f8d69-b7c3-443d-9f54-4185d44bb6cf_isrForCrosstalkSources_1125917_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:12 proc_lsst.multi:146 [INFO] found job 16896476 in provider checkpoint proc_lsst.multi INFO: found job 16896476 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:12 proc_lsst.multi:146 [INFO] found job 16896477 in provider checkpoint proc_lsst.multi INFO: found job 16896477 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e44055ce9933214b706fe3e6bbed3e1 with result from task 2750 parsl.dataflow.dflow DEBUG: Task 2750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/2503599a-0c6b-432e-9f53-be9552cefdf1_isrForCrosstalkSources_1125922_55.stdout parsl.dataflow.dflow INFO: Standard error for task 2750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/2503599a-0c6b-432e-9f53-be9552cefdf1_isrForCrosstalkSources_1125922_55.stderr parsl.dataflow.dflow INFO: Task 2768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e497cd8cc6fcaf473b5b78ffa3c0576 with result from task 2768 parsl.dataflow.dflow DEBUG: Task 2768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2d51edac-ae6c-47ad-b0f5-92d4eaf99a24_isrForCrosstalkSources_1125933_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2d51edac-ae6c-47ad-b0f5-92d4eaf99a24_isrForCrosstalkSources_1125933_28.stderr parsl.dataflow.dflow INFO: Task 2792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db6d68d6b5b330c58f8abf8b5a929f34 with result from task 2792 parsl.dataflow.dflow DEBUG: Task 2792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/c6d58860-543d-4ea4-bd17-a6fb697d2290_isrForCrosstalkSources_1125939_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/c6d58860-543d-4ea4-bd17-a6fb697d2290_isrForCrosstalkSources_1125939_18.stderr parsl.dataflow.dflow INFO: Task 2977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d778bb9adfe23e1a2aae40b9debbc5ed with result from task 2977 parsl.dataflow.dflow DEBUG: Task 2977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a1a38c95-93c8-41a7-9c3d-a3d0fe660c0c_isrForCrosstalkSources_1125909_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a1a38c95-93c8-41a7-9c3d-a3d0fe660c0c_isrForCrosstalkSources_1125909_20.stderr parsl.dataflow.dflow INFO: Task 2966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6464ec0661b3f011fb95e87331b08d3c with result from task 2966 parsl.dataflow.dflow DEBUG: Task 2966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/022a1b2f-d294-4449-b1de-160bd8d55706_isrForCrosstalkSources_1125942_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/022a1b2f-d294-4449-b1de-160bd8d55706_isrForCrosstalkSources_1125942_11.stderr parsl.dataflow.dflow INFO: Task 2879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f5ba6b596c93ade6decb820e3e61881 with result from task 2879 parsl.dataflow.dflow DEBUG: Task 2879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/55d47737-9b9e-4eb9-8005-1a260c0f22af_isrForCrosstalkSources_1125909_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/55d47737-9b9e-4eb9-8005-1a260c0f22af_isrForCrosstalkSources_1125909_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:12 proc_lsst.multi:146 [INFO] found job 16896478 in provider checkpoint proc_lsst.multi INFO: found job 16896478 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfa108f9273c284f72b2a591546a1341 with result from task 2670 parsl.dataflow.dflow DEBUG: Task 2670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a0ea32da-4ed6-41e1-8c62-98f0af8b475d_isrForCrosstalkSources_1125945_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a0ea32da-4ed6-41e1-8c62-98f0af8b475d_isrForCrosstalkSources_1125945_18.stderr parsl.dataflow.dflow INFO: Task 2647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1e90fca251b55bb60388be0922511de with result from task 2647 parsl.dataflow.dflow DEBUG: Task 2647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/4185f51e-72bd-439f-925e-7aced67f193d_isrForCrosstalkSources_1125946_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/4185f51e-72bd-439f-925e-7aced67f193d_isrForCrosstalkSources_1125946_28.stderr parsl.dataflow.dflow INFO: Task 2682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a441b12e89bccf8d5fb8a9a0f53cdba with result from task 2682 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:12 proc_lsst.multi:146 [INFO] found job 16896479 in provider checkpoint proc_lsst.multi INFO: found job 16896479 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/4ecd1ce1-8879-4f0a-93d1-e899c0555e3c_isrForCrosstalkSources_1125899_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/4ecd1ce1-8879-4f0a-93d1-e899c0555e3c_isrForCrosstalkSources_1125899_52.stderr parsl.dataflow.dflow INFO: Task 2968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e26e6750e5afc857cac9a1faf071c0fd with result from task 2968 parsl.dataflow.dflow DEBUG: Task 2968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/6616945f-f743-4df5-969c-deb674a2a990_isrForCrosstalkSources_1125947_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/6616945f-f743-4df5-969c-deb674a2a990_isrForCrosstalkSources_1125947_20.stderr parsl.dataflow.dflow INFO: Task 2988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a56d457cae8ad4474b7c813e270e8c3a with result from task 2988 parsl.dataflow.dflow DEBUG: Task 2988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/e3c71953-d598-4e44-be4f-84dffeb22728_isrForCrosstalkSources_1125908_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/e3c71953-d598-4e44-be4f-84dffeb22728_isrForCrosstalkSources_1125908_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:12 proc_lsst.multi:146 [INFO] found job 16896480 in provider checkpoint proc_lsst.multi INFO: found job 16896480 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8da277c971246380fd5a0f6fcd03b87f with result from task 2914 parsl.dataflow.dflow DEBUG: Task 2914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/0f43f223-4859-4b84-8752-5bbf5c5bdcad_isrForCrosstalkSources_1125947_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/0f43f223-4859-4b84-8752-5bbf5c5bdcad_isrForCrosstalkSources_1125947_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:13 proc_lsst.multi:146 [INFO] found job 16896481 in provider checkpoint proc_lsst.multi INFO: found job 16896481 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e675d351561ec0dea990788deaa45a1f with result from task 2909 parsl.dataflow.dflow DEBUG: Task 2909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/1777e3e6-653e-49b5-8062-44ce3525eb1a_isrForCrosstalkSources_1125912_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/1777e3e6-653e-49b5-8062-44ce3525eb1a_isrForCrosstalkSources_1125912_6.stderr parsl.dataflow.dflow INFO: Task 2887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2033b12d9f18e762af042878db969533 with result from task 2887 parsl.dataflow.dflow DEBUG: Task 2887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/0ff04c2e-fc01-45d8-a6b3-ddd3823e49d9_isrForCrosstalkSources_1125926_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/0ff04c2e-fc01-45d8-a6b3-ddd3823e49d9_isrForCrosstalkSources_1125926_59.stderr parsl.dataflow.dflow INFO: Task 2940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9171719a99fb89b7c5aa9422a098828a with result from task 2940 parsl.dataflow.dflow DEBUG: Task 2940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/acbcd238-3ef1-4fca-9fbf-daff73eb3a98_isrForCrosstalkSources_1125900_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/acbcd238-3ef1-4fca-9fbf-daff73eb3a98_isrForCrosstalkSources_1125900_14.stderr parsl.dataflow.dflow INFO: Task 2942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c2144597d9d283e0708fa0595647dc3 with result from task 2942 parsl.dataflow.dflow DEBUG: Task 2942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/9641f867-2960-42d9-8897-fb2288d7a22c_isrForCrosstalkSources_1125913_22.stdout parsl.dataflow.dflow INFO: Standard error for task 2942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/9641f867-2960-42d9-8897-fb2288d7a22c_isrForCrosstalkSources_1125913_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:13 proc_lsst.multi:146 [INFO] found job 16896482 in provider checkpoint proc_lsst.multi INFO: found job 16896482 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 099e86df00eb6cde7ad3f007c290cb52 with result from task 2880 parsl.dataflow.dflow DEBUG: Task 2880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/a139a535-8055-4dc8-a004-83dd7b654a25_isrForCrosstalkSources_1125932_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/a139a535-8055-4dc8-a004-83dd7b654a25_isrForCrosstalkSources_1125932_25.stderr parsl.dataflow.dflow INFO: Task 2883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ada39ea95e2177356f39bbfa20fcb51c with result from task 2883 parsl.dataflow.dflow DEBUG: Task 2883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/b1d993ea-f1aa-4522-9b8f-24e764be4f59_isrForCrosstalkSources_1125935_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/b1d993ea-f1aa-4522-9b8f-24e764be4f59_isrForCrosstalkSources_1125935_56.stderr parsl.dataflow.dflow INFO: Task 2893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 531f8395ff01c0bcda172c73b756fa84 with result from task 2893 parsl.dataflow.dflow DEBUG: Task 2893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/49505597-b35a-4e0f-a89a-b35f6c878822_isrForCrosstalkSources_1125924_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/49505597-b35a-4e0f-a89a-b35f6c878822_isrForCrosstalkSources_1125924_45.stderr parsl.dataflow.dflow INFO: Task 2971 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a456569229f0f826e4538d05fad0154 with result from task 2971 parsl.dataflow.dflow DEBUG: Task 2971 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/70876277-f40f-492c-86af-1cc28f862fe2_isrForCrosstalkSources_1125921_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/70876277-f40f-492c-86af-1cc28f862fe2_isrForCrosstalkSources_1125921_25.stderr parsl.dataflow.dflow INFO: Task 2920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c2961f685fd05405bd0d9ca49920d7a with result from task 2920 parsl.dataflow.dflow DEBUG: Task 2920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/e8bad1f4-61ce-4824-8b6c-1471712b378b_isrForCrosstalkSources_1125917_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/e8bad1f4-61ce-4824-8b6c-1471712b378b_isrForCrosstalkSources_1125917_10.stderr parsl.dataflow.dflow INFO: Task 2732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f850e11917ffdfd3a938367fde9f25c6 with result from task 2732 parsl.dataflow.dflow DEBUG: Task 2732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/863ebf22-c437-4a35-b035-c56f4c10a558_isrForCrosstalkSources_1125928_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/863ebf22-c437-4a35-b035-c56f4c10a558_isrForCrosstalkSources_1125928_60.stderr parsl.dataflow.dflow INFO: Task 2894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa4bad756c448b38e2afa4afd39e0968 with result from task 2894 parsl.dataflow.dflow DEBUG: Task 2894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/517af224-5c61-4310-b89c-790e52434017_isrForCrosstalkSources_1125934_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/517af224-5c61-4310-b89c-790e52434017_isrForCrosstalkSources_1125934_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:13 proc_lsst.multi:146 [INFO] found job 16896483 in provider checkpoint proc_lsst.multi INFO: found job 16896483 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b0011012fc21478fae075ef2df9932e with result from task 2934 parsl.dataflow.dflow DEBUG: Task 2934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/2f432a00-4f8f-4f44-840f-a08aefc3fede_isrForCrosstalkSources_1125922_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/2f432a00-4f8f-4f44-840f-a08aefc3fede_isrForCrosstalkSources_1125922_48.stderr parsl.dataflow.dflow INFO: Task 2992 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7da9cef402db5031b7252b15c2cdd0d with result from task 2992 parsl.dataflow.dflow DEBUG: Task 2992 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/178d79aa-c80c-407c-b8c9-879d9f77f59e_isrForCrosstalkSources_1125900_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/178d79aa-c80c-407c-b8c9-879d9f77f59e_isrForCrosstalkSources_1125900_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:13 proc_lsst.multi:146 [INFO] found job 16896484 in provider checkpoint proc_lsst.multi INFO: found job 16896484 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 489aaf59326904797cd253e393180055 with result from task 2731 parsl.dataflow.dflow DEBUG: Task 2731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/7d33f64e-8eb4-47df-afe5-24d19f0c9807_isrForCrosstalkSources_1125912_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/7d33f64e-8eb4-47df-afe5-24d19f0c9807_isrForCrosstalkSources_1125912_61.stderr parsl.dataflow.dflow INFO: Task 2896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bb120c074271626fc283b8f1ad89b74 with result from task 2896 parsl.dataflow.dflow DEBUG: Task 2896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/1e0a29a6-7d9e-4b0f-a440-530fbf502033_isrForCrosstalkSources_1125923_49.stdout parsl.dataflow.dflow INFO: Standard error for task 2896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/1e0a29a6-7d9e-4b0f-a440-530fbf502033_isrForCrosstalkSources_1125923_49.stderr parsl.dataflow.dflow INFO: Task 2895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44988f2641c4771364f38c9243fe2c4d with result from task 2895 parsl.dataflow.dflow DEBUG: Task 2895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/52c1fab9-86e6-4188-af84-ade5e79e6c59_isrForCrosstalkSources_1125939_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/52c1fab9-86e6-4188-af84-ade5e79e6c59_isrForCrosstalkSources_1125939_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:14 proc_lsst.multi:146 [INFO] found job 16896485 in provider checkpoint proc_lsst.multi INFO: found job 16896485 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b654a7955ccf2b3d44969fe2e3d257c7 with result from task 2979 parsl.dataflow.dflow DEBUG: Task 2979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/c4d456b4-89c8-4d8d-a5a7-4c9dd9836cde_isrForCrosstalkSources_1125930_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/c4d456b4-89c8-4d8d-a5a7-4c9dd9836cde_isrForCrosstalkSources_1125930_23.stderr parsl.dataflow.dflow INFO: Task 2926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40b00474f136f05ca55d4e1b50b48bd1 with result from task 2926 parsl.dataflow.dflow DEBUG: Task 2926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/1b564fad-9e35-4ebe-bbbe-6b694141c2f9_isrForCrosstalkSources_1125916_60.stdout parsl.dataflow.dflow INFO: Standard error for task 2926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/1b564fad-9e35-4ebe-bbbe-6b694141c2f9_isrForCrosstalkSources_1125916_60.stderr parsl.dataflow.dflow INFO: Task 2961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a78c54a1d0121ea848a6bf305109e5ed with result from task 2961 parsl.dataflow.dflow DEBUG: Task 2961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/6f7fcab6-c24b-4ca0-b260-c6997557542b_isrForCrosstalkSources_1125922_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/6f7fcab6-c24b-4ca0-b260-c6997557542b_isrForCrosstalkSources_1125922_45.stderr parsl.dataflow.dflow INFO: Task 2922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f439b95bbeee595582df7f4f3985f74 with result from task 2922 parsl.dataflow.dflow DEBUG: Task 2922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/bda1bcc6-588e-4111-a751-fe0ec6ca9034_isrForCrosstalkSources_1125898_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/bda1bcc6-588e-4111-a751-fe0ec6ca9034_isrForCrosstalkSources_1125898_13.stderr parsl.dataflow.dflow INFO: Task 2906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c74a5edb813448b03ab40c20177a8d7 with result from task 2906 parsl.dataflow.dflow DEBUG: Task 2906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/acb8d237-f6ae-4e57-834a-04f8064e25e1_isrForCrosstalkSources_1125905_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/acb8d237-f6ae-4e57-834a-04f8064e25e1_isrForCrosstalkSources_1125905_8.stderr parsl.dataflow.dflow INFO: Task 2898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88a885fe5afe0d25c135866a2934570c with result from task 2898 parsl.dataflow.dflow DEBUG: Task 2898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/31c3f5df-2535-4faa-9cf6-573fbec29b5c_isrForCrosstalkSources_1125941_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/31c3f5df-2535-4faa-9cf6-573fbec29b5c_isrForCrosstalkSources_1125941_9.stderr parsl.dataflow.dflow INFO: Task 2917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29b293a4e675ae95a6a7d11baaefa0a1 with result from task 2917 parsl.dataflow.dflow DEBUG: Task 2917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/bf5ddc03-1ae9-4fa4-ae5c-629d175d5b8d_isrForCrosstalkSources_1125914_50.stdout parsl.dataflow.dflow INFO: Standard error for task 2917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/bf5ddc03-1ae9-4fa4-ae5c-629d175d5b8d_isrForCrosstalkSources_1125914_50.stderr parsl.dataflow.dflow INFO: Task 2889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36325479f3a0bcee4ffc9b3bc5493728 with result from task 2889 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:14 proc_lsst.multi:146 [INFO] found job 16896486 in provider checkpoint proc_lsst.multi INFO: found job 16896486 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1841d4f0-3d33-4481-983a-96e495883490_isrForCrosstalkSources_1125900_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1841d4f0-3d33-4481-983a-96e495883490_isrForCrosstalkSources_1125900_15.stderr parsl.dataflow.dflow INFO: Task 2949 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ef416fb85062003dec4b1d8d8005fe6 with result from task 2949 parsl.dataflow.dflow DEBUG: Task 2949 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a7ba0ffa-591d-4db6-a0ea-b90cd2f34b55_isrForCrosstalkSources_1125937_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/a7ba0ffa-591d-4db6-a0ea-b90cd2f34b55_isrForCrosstalkSources_1125937_26.stderr parsl.dataflow.dflow INFO: Task 2884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8512e89e944a8b900c7a8ac3d4058ce6 with result from task 2884 parsl.dataflow.dflow DEBUG: Task 2884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/b0ff2c2b-b6b3-4e34-b5a0-a8f0d63086cc_isrForCrosstalkSources_1125895_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/b0ff2c2b-b6b3-4e34-b5a0-a8f0d63086cc_isrForCrosstalkSources_1125895_26.stderr parsl.dataflow.dflow INFO: Task 2886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87f9e542ae702e8a572a16a6a60fd0df with result from task 2886 parsl.dataflow.dflow DEBUG: Task 2886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/0f29c58c-7d9f-4b6f-9d53-bb966b273d21_isrForCrosstalkSources_1125943_25.stdout parsl.dataflow.dflow INFO: Standard error for task 2886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/0f29c58c-7d9f-4b6f-9d53-bb966b273d21_isrForCrosstalkSources_1125943_25.stderr parsl.dataflow.dflow INFO: Task 2921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29684d2c8b2ba60b80c46ebc7f0481c9 with result from task 2921 parsl.dataflow.dflow DEBUG: Task 2921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/067e8cb4-51f9-4a56-9420-fb6ccc520636_isrForCrosstalkSources_1125907_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/067e8cb4-51f9-4a56-9420-fb6ccc520636_isrForCrosstalkSources_1125907_26.stderr parsl.dataflow.dflow INFO: Task 2925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed23fb322c8fe6ad5bac1860f8a44912 with result from task 2925 parsl.dataflow.dflow DEBUG: Task 2925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/9788cf66-1efb-4588-920c-2eaae3bc8c8a_isrForCrosstalkSources_1125925_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/9788cf66-1efb-4588-920c-2eaae3bc8c8a_isrForCrosstalkSources_1125925_4.stderr parsl.dataflow.dflow INFO: Task 2938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abc016debc744e91e5e1e07639b95cfa with result from task 2938 parsl.dataflow.dflow DEBUG: Task 2938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/c9f5923d-d067-4e8b-a0dc-62e02402c483_isrForCrosstalkSources_1125901_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/c9f5923d-d067-4e8b-a0dc-62e02402c483_isrForCrosstalkSources_1125901_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:14 proc_lsst.multi:146 [INFO] found job 16896487 in provider checkpoint proc_lsst.multi INFO: found job 16896487 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 205d407d0384ad4272e68ea17aa85067 with result from task 2935 parsl.dataflow.dflow DEBUG: Task 2935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/ee1c7d7e-2df7-46dc-839e-9c57ef9ca1f0_isrForCrosstalkSources_1125923_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/ee1c7d7e-2df7-46dc-839e-9c57ef9ca1f0_isrForCrosstalkSources_1125923_47.stderr parsl.dataflow.dflow INFO: Task 2978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f66c32d9fd55cd5f78232e1f047b297 with result from task 2978 parsl.dataflow.dflow DEBUG: Task 2978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/11d2c61b-fd76-4241-bd0a-af9e01236766_isrForCrosstalkSources_1125947_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/11d2c61b-fd76-4241-bd0a-af9e01236766_isrForCrosstalkSources_1125947_48.stderr parsl.dataflow.dflow INFO: Task 2963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2301d5f3939a49d0dd36a5e384ed1410 with result from task 2963 parsl.dataflow.dflow DEBUG: Task 2963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/96f0bd27-2ccf-4db4-bdfa-275b99e7cb83_isrForCrosstalkSources_1125927_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/96f0bd27-2ccf-4db4-bdfa-275b99e7cb83_isrForCrosstalkSources_1125927_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:15 proc_lsst.multi:146 [INFO] found job 16896488 in provider checkpoint proc_lsst.multi INFO: found job 16896488 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcbaa0b0bfba6c122ee79c01ee1fe22c with result from task 2871 parsl.dataflow.dflow DEBUG: Task 2871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/ba7af326-ed82-4f92-a037-fc507024dbbf_isrForCrosstalkSources_1125911_16.stdout parsl.dataflow.dflow INFO: Standard error for task 2871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/ba7af326-ed82-4f92-a037-fc507024dbbf_isrForCrosstalkSources_1125911_16.stderr parsl.dataflow.dflow INFO: Task 2919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26611ebd1f8e88d283f49ca6ca971816 with result from task 2919 parsl.dataflow.dflow DEBUG: Task 2919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/982fb279-5b31-4307-9610-44539aca0085_isrForCrosstalkSources_1125901_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/982fb279-5b31-4307-9610-44539aca0085_isrForCrosstalkSources_1125901_31.stderr parsl.dataflow.dflow INFO: Task 2980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d42a683d57a1b3077c9443428d80a77 with result from task 2980 parsl.dataflow.dflow DEBUG: Task 2980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/20b2c8e6-e767-467c-8f88-5f2e734ea592_isrForCrosstalkSources_1125944_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/20b2c8e6-e767-467c-8f88-5f2e734ea592_isrForCrosstalkSources_1125944_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:15 proc_lsst.multi:146 [INFO] found job 16896489 in provider checkpoint proc_lsst.multi INFO: found job 16896489 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f50724b7f9fe267668ceb850f2b2602c with result from task 2965 parsl.dataflow.dflow DEBUG: Task 2965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/a96dad51-66e3-4094-819b-694de0ffb483_isrForCrosstalkSources_1125913_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/a96dad51-66e3-4094-819b-694de0ffb483_isrForCrosstalkSources_1125913_24.stderr parsl.dataflow.dflow INFO: Task 3046 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8b91f72003605e514f09efabd6e53d9 with result from task 3046 parsl.dataflow.dflow DEBUG: Task 3046 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/cbb6e0a6-e401-4982-86c1-bc6c75540db9_isrForCrosstalkSources_1125899_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/cbb6e0a6-e401-4982-86c1-bc6c75540db9_isrForCrosstalkSources_1125899_20.stderr parsl.dataflow.dflow INFO: Task 2924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29cdb1775842a155f2acdd0f9c870e40 with result from task 2924 parsl.dataflow.dflow DEBUG: Task 2924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8abf87ac-7791-45f0-b7be-308918414948_isrForCrosstalkSources_1125891_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/8abf87ac-7791-45f0-b7be-308918414948_isrForCrosstalkSources_1125891_37.stderr parsl.dataflow.dflow INFO: Task 2881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9b81ce6454a1fd4e0766898e787a0e5 with result from task 2881 parsl.dataflow.dflow DEBUG: Task 2881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/b81ac122-0fb4-44e9-b9b5-584c83a39119_isrForCrosstalkSources_1125893_11.stdout parsl.dataflow.dflow INFO: Standard error for task 2881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/b81ac122-0fb4-44e9-b9b5-584c83a39119_isrForCrosstalkSources_1125893_11.stderr parsl.dataflow.dflow INFO: Task 2995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd639630a9253f5e558bc6dcc38b37b8 with result from task 2995 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:15 proc_lsst.multi:146 [INFO] found job 16896490 in provider checkpoint proc_lsst.multi INFO: found job 16896490 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/1d1bb996-feb4-4bfc-b527-1d55236519a1_isrForCrosstalkSources_1125947_24.stdout parsl.dataflow.dflow INFO: Standard error for task 2995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/1d1bb996-feb4-4bfc-b527-1d55236519a1_isrForCrosstalkSources_1125947_24.stderr parsl.dataflow.dflow INFO: Task 2969 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18bb864b64b5eec8bad4f787c2a0ea76 with result from task 2969 parsl.dataflow.dflow DEBUG: Task 2969 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f60651db-6714-4b39-ba8f-4728c3f2ff4f_isrForCrosstalkSources_1125910_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f60651db-6714-4b39-ba8f-4728c3f2ff4f_isrForCrosstalkSources_1125910_27.stderr parsl.dataflow.dflow INFO: Task 2982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63bfd3df052f15d17a6f97f8b7f1fccf with result from task 2982 parsl.dataflow.dflow DEBUG: Task 2982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/c66802b8-2792-4cef-8f1c-b849787cb1d3_isrForCrosstalkSources_1125890_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/c66802b8-2792-4cef-8f1c-b849787cb1d3_isrForCrosstalkSources_1125890_17.stderr parsl.dataflow.dflow INFO: Task 2890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9694228f4ab785168a1f1bd0674fac1e with result from task 2890 parsl.dataflow.dflow DEBUG: Task 2890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/daf09fbb-9127-4d24-aaf5-d25509fd4f4b_isrForCrosstalkSources_1125888_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/daf09fbb-9127-4d24-aaf5-d25509fd4f4b_isrForCrosstalkSources_1125888_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 2904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af90666afb3e4144a2182cbfe5c93365 with result from task 2904 parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:15 proc_lsst.multi:146 [INFO] found job 16896491 in provider checkpoint proc_lsst.multi INFO: found job 16896491 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/df68f3a7-5701-4311-bcfb-749e17dfa000_isrForCrosstalkSources_1125922_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/df68f3a7-5701-4311-bcfb-749e17dfa000_isrForCrosstalkSources_1125922_19.stderr parsl.dataflow.dflow INFO: Task 2869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58cb8a14b9b98c5ab2bb766ed03500de with result from task 2869 parsl.dataflow.dflow DEBUG: Task 2869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/0e73b55a-33e0-4abf-b6c9-6ae636c8a27d_isrForCrosstalkSources_1125947_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/0e73b55a-33e0-4abf-b6c9-6ae636c8a27d_isrForCrosstalkSources_1125947_13.stderr parsl.dataflow.dflow INFO: Task 2989 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5909aaefb452d92a8ae23116cd9ef812 with result from task 2989 parsl.dataflow.dflow DEBUG: Task 2989 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/fe5e7f6d-232a-4872-bda9-7969cece15b1_isrForCrosstalkSources_1125923_23.stdout parsl.dataflow.dflow INFO: Standard error for task 2989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/fe5e7f6d-232a-4872-bda9-7969cece15b1_isrForCrosstalkSources_1125923_23.stderr parsl.dataflow.dflow INFO: Task 2986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6950a28cfff9e47db15a8e177366bc7a with result from task 2986 parsl.dataflow.dflow DEBUG: Task 2986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/57d9de57-cdd5-417a-acae-58b7acd76d97_isrForCrosstalkSources_1125938_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/57d9de57-cdd5-417a-acae-58b7acd76d97_isrForCrosstalkSources_1125938_47.stderr parsl.dataflow.dflow INFO: Task 2862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca58c864926692f9b58ff3656b87733e with result from task 2862 parsl.dataflow.dflow DEBUG: Task 2862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/15e16e1e-dcd4-4f7a-a74e-86f487243c95_isrForCrosstalkSources_1125891_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/15e16e1e-dcd4-4f7a-a74e-86f487243c95_isrForCrosstalkSources_1125891_42.stderr parsl.dataflow.dflow INFO: Task 2897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4e79bad48d2baf5691d16e5cbdcbc41 with result from task 2897 parsl.dataflow.dflow DEBUG: Task 2897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/8aa8e045-368d-4133-a968-ee86c5ac2efa_isrForCrosstalkSources_1125917_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/8aa8e045-368d-4133-a968-ee86c5ac2efa_isrForCrosstalkSources_1125917_31.stderr parsl.dataflow.dflow INFO: Task 3009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 103cbfbf926f292c60f2fc6d2a54b3a7 with result from task 3009 parsl.dataflow.dflow DEBUG: Task 3009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/18bdd824-3fca-43e8-9a8e-3a2a54cfba6a_isrForCrosstalkSources_1125908_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/18bdd824-3fca-43e8-9a8e-3a2a54cfba6a_isrForCrosstalkSources_1125908_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:15 proc_lsst.multi:146 [INFO] found job 16896492 in provider checkpoint proc_lsst.multi INFO: found job 16896492 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37928195b1fad719d207fde296aebe45 with result from task 2956 parsl.dataflow.dflow DEBUG: Task 2956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/1aed101f-cd5e-450d-93fe-3d1ddc53b0d1_isrForCrosstalkSources_1125907_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/1aed101f-cd5e-450d-93fe-3d1ddc53b0d1_isrForCrosstalkSources_1125907_37.stderr parsl.dataflow.dflow INFO: Task 3055 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa694e170c67bc11edb4e7fdc99d69cf with result from task 3055 parsl.dataflow.dflow DEBUG: Task 3055 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/b8caaef0-fad4-4bc9-9ef1-b6146dcfbba6_isrForCrosstalkSources_1125912_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/b8caaef0-fad4-4bc9-9ef1-b6146dcfbba6_isrForCrosstalkSources_1125912_25.stderr parsl.dataflow.dflow INFO: Task 2918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f20e0cefc85c1a53b73d2478961d81cd with result from task 2918 parsl.dataflow.dflow DEBUG: Task 2918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/688e6f1e-6596-41bd-ab73-efa8e5338add_isrForCrosstalkSources_1125909_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/688e6f1e-6596-41bd-ab73-efa8e5338add_isrForCrosstalkSources_1125909_5.stderr parsl.dataflow.dflow INFO: Task 2945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b50d27cc950c91e39860f15b83ae4599 with result from task 2945 parsl.dataflow.dflow DEBUG: Task 2945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/feb3c2fc-1d39-49f2-a994-b26806ef4953_isrForCrosstalkSources_1125908_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/feb3c2fc-1d39-49f2-a994-b26806ef4953_isrForCrosstalkSources_1125908_1.stderr parsl.dataflow.dflow INFO: Task 2957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bcd2d75088430ff879acf2e2b4c966c with result from task 2957 parsl.dataflow.dflow DEBUG: Task 2957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/cee2b05e-d427-47c8-9f10-fdc48d4a7acf_isrForCrosstalkSources_1125914_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/cee2b05e-d427-47c8-9f10-fdc48d4a7acf_isrForCrosstalkSources_1125914_31.stderr parsl.dataflow.dflow INFO: Task 3004 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9ddc25f594f1510a6fb63d5fbd34124 with result from task 3004 parsl.dataflow.dflow DEBUG: Task 3004 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/887331fd-ead6-43cc-9eb8-204029fbdb18_isrForCrosstalkSources_1125918_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/887331fd-ead6-43cc-9eb8-204029fbdb18_isrForCrosstalkSources_1125918_12.stderr parsl.dataflow.dflow INFO: Task 2824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aac72a8d1cefc8abcdd1b73eea03c614 with result from task 2824 parsl.dataflow.dflow DEBUG: Task 2824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/22a2f76a-c1c2-4a99-9b17-5ef696328349_isrForCrosstalkSources_1125921_8.stdout parsl.dataflow.dflow INFO: Standard error for task 2824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/22a2f76a-c1c2-4a99-9b17-5ef696328349_isrForCrosstalkSources_1125921_8.stderr parsl.dataflow.dflow INFO: Task 3025 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b86e6a980d6439483d0aecd910b6223 with result from task 3025 parsl.dataflow.dflow DEBUG: Task 3025 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/317839e1-6ee9-4c96-8a5c-770b9f5ed515_isrForCrosstalkSources_1125904_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/317839e1-6ee9-4c96-8a5c-770b9f5ed515_isrForCrosstalkSources_1125904_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:16 proc_lsst.multi:146 [INFO] found job 16896493 in provider checkpoint proc_lsst.multi INFO: found job 16896493 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3314220312f1b7a1277b3869b0a19a65 with result from task 2959 parsl.dataflow.dflow DEBUG: Task 2959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b0c15585-3673-4e1e-8516-221a1b3c0829_isrForCrosstalkSources_1125934_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/b0c15585-3673-4e1e-8516-221a1b3c0829_isrForCrosstalkSources_1125934_17.stderr parsl.dataflow.dflow INFO: Task 2930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e31d2885178bf63f50a61c6548bb6d01 with result from task 2930 parsl.dataflow.dflow DEBUG: Task 2930 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/091a19c4-5b6d-4fbe-98de-2305d1be7b0a_isrForCrosstalkSources_1125911_17.stdout parsl.dataflow.dflow INFO: Standard error for task 2930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/091a19c4-5b6d-4fbe-98de-2305d1be7b0a_isrForCrosstalkSources_1125911_17.stderr parsl.dataflow.dflow INFO: Task 2929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fe0e5329dac5ffd18050e581b72788f with result from task 2929 parsl.dataflow.dflow DEBUG: Task 2929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/419bb142-85bf-4343-98ad-921145761ece_isrForCrosstalkSources_1125900_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/419bb142-85bf-4343-98ad-921145761ece_isrForCrosstalkSources_1125900_37.stderr parsl.dataflow.dflow INFO: Task 2932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 776824152a412fdc84cfe0eff615b714 with result from task 2932 parsl.dataflow.dflow DEBUG: Task 2932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/47f065b4-0862-4952-b6fe-85769fbc4e66_isrForCrosstalkSources_1125894_1.stdout parsl.dataflow.dflow INFO: Standard error for task 2932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/47f065b4-0862-4952-b6fe-85769fbc4e66_isrForCrosstalkSources_1125894_1.stderr parsl.dataflow.dflow INFO: Task 3024 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccafc03cfcf0cf3e61a55c8372d2431d with result from task 3024 parsl.dataflow.dflow DEBUG: Task 3024 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/3f696ac6-3c0c-40f4-bb00-9fbf8058121e_isrForCrosstalkSources_1125934_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/3f696ac6-3c0c-40f4-bb00-9fbf8058121e_isrForCrosstalkSources_1125934_26.stderr parsl.dataflow.dflow INFO: Task 2944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b54096110f8626ddf4fd153e05287806 with result from task 2944 parsl.dataflow.dflow DEBUG: Task 2944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/c45a72bf-2976-4c4b-bee9-7ceafa1e5697_isrForCrosstalkSources_1125926_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/c45a72bf-2976-4c4b-bee9-7ceafa1e5697_isrForCrosstalkSources_1125926_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 2964 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 3bf4915fd27ca1f51198a08cb52438cd with result from task 2964 parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 2964 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 2964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/fa0ae72a-7391-460c-b50d-1fd1409a08ac_isrForCrosstalkSources_1125919_40.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 2964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/fa0ae72a-7391-460c-b50d-1fd1409a08ac_isrForCrosstalkSources_1125919_40.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:16 proc_lsst.multi:146 [INFO] found job 16896494 in provider checkpoint proc_lsst.multi INFO: found job 16896494 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f31ee21502953ab0cc8bf295c79a2d02 with result from task 2867 parsl.dataflow.dflow DEBUG: Task 2867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6af75b3b-c7fe-4186-a25c-e6fcb5c4ec2a_isrForCrosstalkSources_1125927_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6af75b3b-c7fe-4186-a25c-e6fcb5c4ec2a_isrForCrosstalkSources_1125927_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:16 proc_lsst.multi:146 [INFO] found job 16896495 in provider checkpoint proc_lsst.multi INFO: found job 16896495 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a1dd66cbfeb61eaac5fee5304d2c2cf with result from task 2847 parsl.dataflow.dflow DEBUG: Task 2847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/5fdaa94e-e8c9-4839-a4d9-178df6523665_isrForCrosstalkSources_1125896_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/5fdaa94e-e8c9-4839-a4d9-178df6523665_isrForCrosstalkSources_1125896_56.stderr parsl.dataflow.dflow INFO: Task 2939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b484810a103793f8a2acdb61341fd32 with result from task 2939 parsl.dataflow.dflow DEBUG: Task 2939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/5b913757-d4ea-49b2-99e2-eb5fdc646b6c_isrForCrosstalkSources_1125927_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/5b913757-d4ea-49b2-99e2-eb5fdc646b6c_isrForCrosstalkSources_1125927_53.stderr parsl.dataflow.dflow INFO: Task 2952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dd977e3d7c7fb31b44914f1aa34d0bd with result from task 2952 parsl.dataflow.dflow DEBUG: Task 2952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/898940c9-90cb-43a7-9d63-d9b4df39fbda_isrForCrosstalkSources_1125920_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/898940c9-90cb-43a7-9d63-d9b4df39fbda_isrForCrosstalkSources_1125920_51.stderr parsl.dataflow.dflow INFO: Task 2958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af3c550c91cec7c3f7e355d14b01fe50 with result from task 2958 parsl.dataflow.dflow DEBUG: Task 2958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/6a10bad2-fbaf-4d92-a61b-000d024e8c98_isrForCrosstalkSources_1125917_27.stdout parsl.dataflow.dflow INFO: Standard error for task 2958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/6a10bad2-fbaf-4d92-a61b-000d024e8c98_isrForCrosstalkSources_1125917_27.stderr parsl.dataflow.dflow INFO: Task 2903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8acda4668c4aa01dc7cf42246947bb9 with result from task 2903 parsl.dataflow.dflow DEBUG: Task 2903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/cb9d45df-464c-42c0-a016-b0dcfbdc028d_isrForCrosstalkSources_1125910_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/cb9d45df-464c-42c0-a016-b0dcfbdc028d_isrForCrosstalkSources_1125910_19.stderr parsl.dataflow.dflow INFO: Task 3032 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2075d69f0036a9061a7cc207e900cb13 with result from task 3032 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:16 proc_lsst.multi:146 [INFO] found job 16896496 in provider checkpoint proc_lsst.multi INFO: found job 16896496 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 3032 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/dabdb0fb-6a27-4e2a-b79c-4aa2e0bac857_isrForCrosstalkSources_1125909_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/dabdb0fb-6a27-4e2a-b79c-4aa2e0bac857_isrForCrosstalkSources_1125909_14.stderr parsl.dataflow.dflow INFO: Task 2902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8aa555d98cf9240ae3699296d4cb160f with result from task 2902 parsl.dataflow.dflow DEBUG: Task 2902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/81e5435c-b020-4c30-85e5-b99e608a371d_isrForCrosstalkSources_1125937_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/81e5435c-b020-4c30-85e5-b99e608a371d_isrForCrosstalkSources_1125937_29.stderr parsl.dataflow.dflow INFO: Task 2931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2a0ce9a79f8f1fa7ab62a886c4feb8c with result from task 2931 parsl.dataflow.dflow DEBUG: Task 2931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f97ca5f1-f1c2-4a7b-af7a-6e1c4e5984f0_isrForCrosstalkSources_1125897_53.stdout parsl.dataflow.dflow INFO: Standard error for task 2931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f97ca5f1-f1c2-4a7b-af7a-6e1c4e5984f0_isrForCrosstalkSources_1125897_53.stderr parsl.dataflow.dflow INFO: Task 2870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fa37f56a4cc8847d092ce9b819a1371 with result from task 2870 parsl.dataflow.dflow DEBUG: Task 2870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/f0d5cd0b-b5f6-45ec-8ace-b276603ea8df_isrForCrosstalkSources_1125941_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/f0d5cd0b-b5f6-45ec-8ace-b276603ea8df_isrForCrosstalkSources_1125941_2.stderr parsl.dataflow.dflow INFO: Task 2916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa63dd6df665b7223daa1b4286f0d389 with result from task 2916 parsl.dataflow.dflow DEBUG: Task 2916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/dee6e409-3b86-44fa-a7a5-3940f58250d0_isrForCrosstalkSources_1125894_44.stdout parsl.dataflow.dflow INFO: Standard error for task 2916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/dee6e409-3b86-44fa-a7a5-3940f58250d0_isrForCrosstalkSources_1125894_44.stderr parsl.dataflow.dflow INFO: Task 2915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97210851beec8015dbfde8b8b8755a1e with result from task 2915 parsl.dataflow.dflow DEBUG: Task 2915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/49209465-c317-4587-9405-3565e3e06c87_isrForCrosstalkSources_1125911_38.stdout parsl.dataflow.dflow INFO: Standard error for task 2915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/49209465-c317-4587-9405-3565e3e06c87_isrForCrosstalkSources_1125911_38.stderr parsl.dataflow.dflow INFO: Task 2905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e68885b46b96d50d357af7eeb59204d with result from task 2905 parsl.dataflow.dflow DEBUG: Task 2905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/6db1bb92-92a0-4d16-b279-32ff005a6ff6_isrForCrosstalkSources_1125900_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/6db1bb92-92a0-4d16-b279-32ff005a6ff6_isrForCrosstalkSources_1125900_34.stderr parsl.dataflow.dflow INFO: Task 2912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93b537462d8602bfa45372c26d95e864 with result from task 2912 parsl.dataflow.dflow DEBUG: Task 2912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/d504f3fe-c496-4e5a-aa1d-369ce91be5c8_isrForCrosstalkSources_1125930_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/d504f3fe-c496-4e5a-aa1d-369ce91be5c8_isrForCrosstalkSources_1125930_12.stderr parsl.dataflow.dflow INFO: Task 3068 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13e53889250fd0854a4f57b1467ed728 with result from task 3068 parsl.dataflow.dflow DEBUG: Task 3068 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/6c65b762-535d-4d28-baa2-c923ba5f90d2_isrForCrosstalkSources_1125912_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/6c65b762-535d-4d28-baa2-c923ba5f90d2_isrForCrosstalkSources_1125912_42.stderr parsl.dataflow.dflow INFO: Task 3010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf5138e010b16cc33f91f270197b6a73 with result from task 3010 parsl.dataflow.dflow DEBUG: Task 3010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/0627355b-1334-4157-be8c-cdd67685aef8_isrForCrosstalkSources_1125903_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/0627355b-1334-4157-be8c-cdd67685aef8_isrForCrosstalkSources_1125903_60.stderr parsl.dataflow.dflow INFO: Task 2899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 638b10a3d1e6578c56002fd419219338 with result from task 2899 parsl.dataflow.dflow DEBUG: Task 2899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/69475c1c-e938-4fcc-a5cf-f0116c31da3e_isrForCrosstalkSources_1125907_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/69475c1c-e938-4fcc-a5cf-f0116c31da3e_isrForCrosstalkSources_1125907_43.stderr parsl.dataflow.dflow INFO: Task 3014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c092ef45fc2bdeb758f67f8b2f79344 with result from task 3014 parsl.dataflow.dflow DEBUG: Task 3014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/72385140-c694-4a61-88f9-2e04ebb7c697_isrForCrosstalkSources_1125928_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/72385140-c694-4a61-88f9-2e04ebb7c697_isrForCrosstalkSources_1125928_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:17 proc_lsst.multi:146 [INFO] found job 16896497 in provider checkpoint proc_lsst.multi INFO: found job 16896497 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef2e004e22475d5ddba993efbfd53c3f with result from task 2901 parsl.dataflow.dflow DEBUG: Task 2901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/7d939404-1755-41ee-8691-8acb596f492e_isrForCrosstalkSources_1125938_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/7d939404-1755-41ee-8691-8acb596f492e_isrForCrosstalkSources_1125938_20.stderr parsl.dataflow.dflow INFO: Task 3056 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ceb1684134a022be78e206ceee0c9b44 with result from task 3056 parsl.dataflow.dflow DEBUG: Task 3056 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/c3717ad7-24f8-4f47-a364-0f67350a1fb4_isrForCrosstalkSources_1125896_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/c3717ad7-24f8-4f47-a364-0f67350a1fb4_isrForCrosstalkSources_1125896_27.stderr parsl.dataflow.dflow INFO: Task 3088 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22c184e95270b09d0ebd0ec97be230ef with result from task 3088 parsl.dataflow.dflow DEBUG: Task 3088 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/4b976a96-1d24-452e-b40a-1c1b0d3b10db_isrForCrosstalkSources_1125902_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/4b976a96-1d24-452e-b40a-1c1b0d3b10db_isrForCrosstalkSources_1125902_28.stderr parsl.dataflow.dflow INFO: Task 2936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6330e3081dd836a84fc5631f640b032b with result from task 2936 parsl.dataflow.dflow DEBUG: Task 2936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/10d76b01-9b99-4ed0-9616-2d647d878289_isrForCrosstalkSources_1125907_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/10d76b01-9b99-4ed0-9616-2d647d878289_isrForCrosstalkSources_1125907_45.stderr parsl.dataflow.dflow INFO: Task 3026 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46ea05d85593e44a4144c4ae42823cea with result from task 3026 parsl.dataflow.dflow DEBUG: Task 3026 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/86d28b6a-71a6-4750-bd7f-463075c47f2e_isrForCrosstalkSources_1125928_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/86d28b6a-71a6-4750-bd7f-463075c47f2e_isrForCrosstalkSources_1125928_8.stderr parsl.dataflow.dflow INFO: Task 3020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 566bfe5733ffee16ddf3037342121529 with result from task 3020 parsl.dataflow.dflow DEBUG: Task 3020 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/c61a67ac-3b9d-4cad-a97c-9030b77d142e_isrForCrosstalkSources_1125888_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/c61a67ac-3b9d-4cad-a97c-9030b77d142e_isrForCrosstalkSources_1125888_28.stderr parsl.dataflow.dflow INFO: Task 3063 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 321e46d700286c5b555b97823f777d19 with result from task 3063 parsl.dataflow.dflow DEBUG: Task 3063 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/9bd7d441-38fd-413b-9a14-14260539302c_isrForCrosstalkSources_1125901_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/9bd7d441-38fd-413b-9a14-14260539302c_isrForCrosstalkSources_1125901_26.stderr parsl.dataflow.dflow INFO: Task 2967 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8804dabd7d7d246cde8a6267fe9b0220 with result from task 2967 parsl.dataflow.dflow DEBUG: Task 2967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/7a5f46c7-11a8-42aa-8be8-15d7c0bd43e4_isrForCrosstalkSources_1125912_20.stdout parsl.dataflow.dflow INFO: Standard error for task 2967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/7a5f46c7-11a8-42aa-8be8-15d7c0bd43e4_isrForCrosstalkSources_1125912_20.stderr parsl.dataflow.dflow INFO: Task 2951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d8bffe97d92f5bbd5100ddf2e6f4b16 with result from task 2951 parsl.dataflow.dflow DEBUG: Task 2951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/67595fff-92fe-419a-aecb-34e71d580c47_isrForCrosstalkSources_1125934_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/67595fff-92fe-419a-aecb-34e71d580c47_isrForCrosstalkSources_1125934_29.stderr parsl.dataflow.dflow INFO: Task 2946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 985de1766049d1c83939f3e081e17b2e with result from task 2946 parsl.dataflow.dflow DEBUG: Task 2946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/deacee88-e1ed-4f34-b9f1-8ddeb1201f4b_isrForCrosstalkSources_1125896_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/deacee88-e1ed-4f34-b9f1-8ddeb1201f4b_isrForCrosstalkSources_1125896_34.stderr parsl.dataflow.dflow INFO: Task 2943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 864fc191967d4204c4822ee34238b2a9 with result from task 2943 parsl.dataflow.dflow DEBUG: Task 2943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/1e41ce34-936d-41d1-94de-14968589ca8b_isrForCrosstalkSources_1125889_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/1e41ce34-936d-41d1-94de-14968589ca8b_isrForCrosstalkSources_1125889_42.stderr parsl.dataflow.dflow INFO: Task 2911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fbe0429c16d0ebb4b52b62714c613de with result from task 2911 parsl.dataflow.dflow DEBUG: Task 2911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/dc13f9bf-36a1-4df9-b111-5adec22948cd_isrForCrosstalkSources_1125908_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/dc13f9bf-36a1-4df9-b111-5adec22948cd_isrForCrosstalkSources_1125908_57.stderr parsl.dataflow.dflow INFO: Task 3067 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9936ccc4a836fa00250c141538fc9708 with result from task 3067 parsl.dataflow.dflow DEBUG: Task 3067 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/82006100-599d-436f-9240-0e7ae7a3be98_isrForCrosstalkSources_1125896_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/82006100-599d-436f-9240-0e7ae7a3be98_isrForCrosstalkSources_1125896_25.stderr parsl.dataflow.dflow INFO: Task 3042 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 953622437cb6bf0f8e4b20d77a2edad7 with result from task 3042 parsl.dataflow.dflow DEBUG: Task 3042 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/bb1be7e1-1fd3-4f11-9a06-2a0b19a08138_isrForCrosstalkSources_1125890_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/bb1be7e1-1fd3-4f11-9a06-2a0b19a08138_isrForCrosstalkSources_1125890_50.stderr parsl.dataflow.dflow INFO: Task 2842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab225eb32bd6148a508ee452fa387b84 with result from task 2842 parsl.dataflow.dflow DEBUG: Task 2842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/477571b4-a616-41f5-bcb7-140c10224880_isrForCrosstalkSources_1125910_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/477571b4-a616-41f5-bcb7-140c10224880_isrForCrosstalkSources_1125910_28.stderr parsl.dataflow.dflow INFO: Task 3072 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c59256d1d35e8bff3b13efc49b0f94c with result from task 3072 parsl.dataflow.dflow DEBUG: Task 3072 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/44d03f63-af1b-4744-b2ac-ea146518a009_isrForCrosstalkSources_1125943_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/44d03f63-af1b-4744-b2ac-ea146518a009_isrForCrosstalkSources_1125943_54.stderr parsl.dataflow.dflow INFO: Task 3107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3afc3af3055d4a663c162f8de965b2f8 with result from task 3107 parsl.dataflow.dflow DEBUG: Task 3107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/b365f0f0-8555-4d7d-8244-9fdbfae08e72_isrForCrosstalkSources_1125902_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/b365f0f0-8555-4d7d-8244-9fdbfae08e72_isrForCrosstalkSources_1125902_26.stderr parsl.dataflow.dflow INFO: Task 2907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1075bedc5feef3553dc0d93acf362b7f with result from task 2907 parsl.dataflow.dflow DEBUG: Task 2907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/ec04d94f-a830-4d9f-85f3-9c195aa4cb91_isrForCrosstalkSources_1125935_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/ec04d94f-a830-4d9f-85f3-9c195aa4cb91_isrForCrosstalkSources_1125935_48.stderr parsl.dataflow.dflow INFO: Task 3021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c475d21280f4e56bdaa72b856fb567c6 with result from task 3021 parsl.dataflow.dflow DEBUG: Task 3021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/4dbaa53b-2194-4a4b-aedf-dd0e213a875c_isrForCrosstalkSources_1125905_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/4dbaa53b-2194-4a4b-aedf-dd0e213a875c_isrForCrosstalkSources_1125905_16.stderr parsl.dataflow.dflow INFO: Task 3074 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e9457905764f3ec22024eb5268a161c with result from task 3074 parsl.dataflow.dflow DEBUG: Task 3074 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/3bed326d-313e-4699-a82e-871ea5d84569_isrForCrosstalkSources_1125908_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/3bed326d-313e-4699-a82e-871ea5d84569_isrForCrosstalkSources_1125908_46.stderr parsl.dataflow.dflow INFO: Task 3082 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db21a4063c6b26eb6c297ed6994c5d2c with result from task 3082 parsl.dataflow.dflow DEBUG: Task 3082 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e8d27326-be9e-4653-a7ac-cd3a34f9277e_isrForCrosstalkSources_1125901_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e8d27326-be9e-4653-a7ac-cd3a34f9277e_isrForCrosstalkSources_1125901_44.stderr parsl.dataflow.dflow INFO: Task 3070 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe70fd2738aa559b7b82c75ea00bd300 with result from task 3070 parsl.dataflow.dflow DEBUG: Task 3070 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/0c7ce43e-9bef-43b7-aead-542bcdf9dc79_isrForCrosstalkSources_1125944_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/0c7ce43e-9bef-43b7-aead-542bcdf9dc79_isrForCrosstalkSources_1125944_25.stderr parsl.dataflow.dflow INFO: Task 2973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac0ce21c28a85e9f25b8a7691ecc6a0f with result from task 2973 parsl.dataflow.dflow DEBUG: Task 2973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/f803717f-fdb0-4c7b-94d1-1b027948d54e_isrForCrosstalkSources_1125921_58.stdout parsl.dataflow.dflow INFO: Standard error for task 2973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/f803717f-fdb0-4c7b-94d1-1b027948d54e_isrForCrosstalkSources_1125921_58.stderr parsl.dataflow.dflow INFO: Task 3076 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0a7afc3839e66fd8b067b3aac3909a7 with result from task 3076 parsl.dataflow.dflow DEBUG: Task 3076 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8642ae0a-760c-48ad-8a19-fd33115154c0_isrForCrosstalkSources_1125936_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8642ae0a-760c-48ad-8a19-fd33115154c0_isrForCrosstalkSources_1125936_2.stderr parsl.dataflow.dflow INFO: Task 3016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e29dcd465be226598341781b68373607 with result from task 3016 parsl.dataflow.dflow DEBUG: Task 3016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/031d099e-55ec-439e-bb04-269710bebccc_isrForCrosstalkSources_1125945_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/031d099e-55ec-439e-bb04-269710bebccc_isrForCrosstalkSources_1125945_61.stderr parsl.dataflow.dflow INFO: Task 3018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1797a4db5236a7fb204980388547f7f5 with result from task 3018 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:17 proc_lsst.multi:146 [INFO] found job 16896498 in provider checkpoint proc_lsst.multi INFO: found job 16896498 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 3018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/0157886d-693b-4a22-899d-4f0a644428b4_isrForCrosstalkSources_1125915_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/0157886d-693b-4a22-899d-4f0a644428b4_isrForCrosstalkSources_1125915_34.stderr parsl.dataflow.dflow INFO: Task 3031 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e3032a57322d75902e2deed536de977 with result from task 3031 parsl.dataflow.dflow DEBUG: Task 3031 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/da2b9ef5-6bdc-4ee1-be8c-3dd98e78629e_isrForCrosstalkSources_1125893_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/da2b9ef5-6bdc-4ee1-be8c-3dd98e78629e_isrForCrosstalkSources_1125893_48.stderr parsl.dataflow.dflow INFO: Task 3001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a460eac21a37ebcb0766644ac5ca8ad with result from task 3001 parsl.dataflow.dflow DEBUG: Task 3001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/79658052-88bc-49a5-ac25-3b01c2908c4d_isrForCrosstalkSources_1125908_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/79658052-88bc-49a5-ac25-3b01c2908c4d_isrForCrosstalkSources_1125908_34.stderr parsl.dataflow.dflow INFO: Task 2817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 504190f1a6d8ad5785291d2dd1f844bb with result from task 2817 parsl.dataflow.dflow DEBUG: Task 2817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ca7437f2-f9df-40e5-9fe5-7380398f70b3_isrForCrosstalkSources_1125902_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ca7437f2-f9df-40e5-9fe5-7380398f70b3_isrForCrosstalkSources_1125902_43.stderr parsl.dataflow.dflow INFO: Task 2910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 092bca851f76850b26b77954add5c55c with result from task 2910 parsl.dataflow.dflow DEBUG: Task 2910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/2224e72b-eb50-43b6-8c19-4bf52c00bb11_isrForCrosstalkSources_1125928_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/2224e72b-eb50-43b6-8c19-4bf52c00bb11_isrForCrosstalkSources_1125928_31.stderr parsl.dataflow.dflow INFO: Task 2997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce101a91500228a910911ce37786e426 with result from task 2997 parsl.dataflow.dflow DEBUG: Task 2997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/b78e250a-6987-42de-a98a-19c4da0ce26e_isrForCrosstalkSources_1125941_33.stdout parsl.dataflow.dflow INFO: Standard error for task 2997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/b78e250a-6987-42de-a98a-19c4da0ce26e_isrForCrosstalkSources_1125941_33.stderr parsl.dataflow.dflow INFO: Task 2844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecd9aed9c80c9eb14a5f9daa008f771b with result from task 2844 parsl.dataflow.dflow DEBUG: Task 2844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/7318e62d-55c8-40b9-9022-e334c46b125a_isrForCrosstalkSources_1125902_5.stdout parsl.dataflow.dflow INFO: Standard error for task 2844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/7318e62d-55c8-40b9-9022-e334c46b125a_isrForCrosstalkSources_1125902_5.stderr parsl.dataflow.dflow INFO: Task 3075 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e122e6863af684ec30bb9da78cd66922 with result from task 3075 parsl.dataflow.dflow DEBUG: Task 3075 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/2a015193-b520-4208-b592-f470c25e29f8_isrForCrosstalkSources_1125888_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/2a015193-b520-4208-b592-f470c25e29f8_isrForCrosstalkSources_1125888_38.stderr parsl.dataflow.dflow INFO: Task 2954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 685a407cd82b8a137832b3ae363f1bf1 with result from task 2954 parsl.dataflow.dflow DEBUG: Task 2954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/6f0a56f0-46d8-4ab4-8c13-b0d7d5562380_isrForCrosstalkSources_1125935_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/6f0a56f0-46d8-4ab4-8c13-b0d7d5562380_isrForCrosstalkSources_1125935_29.stderr parsl.dataflow.dflow INFO: Task 3103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58931a4268e803b3cb5ada545686fd37 with result from task 3103 parsl.dataflow.dflow DEBUG: Task 3103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/49a43ef3-48c1-4080-8e7a-f9fbb2efb66e_isrForCrosstalkSources_1125896_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/49a43ef3-48c1-4080-8e7a-f9fbb2efb66e_isrForCrosstalkSources_1125896_5.stderr parsl.dataflow.dflow INFO: Task 2993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f59f5175f4f2610ec1bd7d628e1e5973 with result from task 2993 parsl.dataflow.dflow DEBUG: Task 2993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/cf771bea-a263-466f-8def-92fe741ec5d3_isrForCrosstalkSources_1125927_19.stdout parsl.dataflow.dflow INFO: Standard error for task 2993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/cf771bea-a263-466f-8def-92fe741ec5d3_isrForCrosstalkSources_1125927_19.stderr parsl.dataflow.dflow INFO: Task 2985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e51ab9f086b954157e5169ebe1d467a2 with result from task 2985 parsl.dataflow.dflow DEBUG: Task 2985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/01ade9fa-bf85-47e7-b98c-33dd1fc7da45_isrForCrosstalkSources_1125935_37.stdout parsl.dataflow.dflow INFO: Standard error for task 2985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/01ade9fa-bf85-47e7-b98c-33dd1fc7da45_isrForCrosstalkSources_1125935_37.stderr parsl.dataflow.dflow INFO: Task 2818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82e71f9eec093a30662db99c3ce0798c with result from task 2818 parsl.dataflow.dflow DEBUG: Task 2818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/7b863f91-b789-471e-b713-3ff087a3defe_isrForCrosstalkSources_1125944_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/7b863f91-b789-471e-b713-3ff087a3defe_isrForCrosstalkSources_1125944_47.stderr parsl.dataflow.dflow INFO: Task 3013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4c5464c40122ed95be13e486b029acc with result from task 3013 parsl.dataflow.dflow DEBUG: Task 3013 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/172f6b6f-cf3b-4040-aac7-04a4b47057b3_isrForCrosstalkSources_1125924_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/172f6b6f-cf3b-4040-aac7-04a4b47057b3_isrForCrosstalkSources_1125924_18.stderr parsl.dataflow.dflow INFO: Task 2845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 323b1847fa8478aae57e7b587ecf0bc6 with result from task 2845 parsl.dataflow.dflow DEBUG: Task 2845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/a614c514-9dd0-4578-a580-5214ce782c57_isrForCrosstalkSources_1125892_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/a614c514-9dd0-4578-a580-5214ce782c57_isrForCrosstalkSources_1125892_3.stderr parsl.dataflow.dflow INFO: Task 3048 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c62ab84e09f96e938bb5c89e2900ab9f with result from task 3048 parsl.dataflow.dflow DEBUG: Task 3048 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/23dda8d0-33db-450f-bda9-425e29967186_isrForCrosstalkSources_1125898_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/23dda8d0-33db-450f-bda9-425e29967186_isrForCrosstalkSources_1125898_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:18 proc_lsst.multi:146 [INFO] found job 16896499 in provider checkpoint proc_lsst.multi INFO: found job 16896499 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 044dca2656fd36543bc4d51e3e91e8a8 with result from task 3104 parsl.dataflow.dflow DEBUG: Task 3104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/2704ca30-2737-49c8-ba33-2747f5a7cc35_isrForCrosstalkSources_1125892_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/2704ca30-2737-49c8-ba33-2747f5a7cc35_isrForCrosstalkSources_1125892_20.stderr parsl.dataflow.dflow INFO: Task 2950 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1b6292b41b28e6be7f68b35401fb188 with result from task 2950 parsl.dataflow.dflow DEBUG: Task 2950 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/e6435a85-4e91-47eb-a796-2446245cb99d_isrForCrosstalkSources_1125923_7.stdout parsl.dataflow.dflow INFO: Standard error for task 2950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/e6435a85-4e91-47eb-a796-2446245cb99d_isrForCrosstalkSources_1125923_7.stderr parsl.dataflow.dflow INFO: Task 3083 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d391a8deaca11070e8f308cb4185326 with result from task 3083 parsl.dataflow.dflow DEBUG: Task 3083 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/cd1ba6bf-b2e3-4e9a-981a-49a892acc54c_isrForCrosstalkSources_1125922_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/cd1ba6bf-b2e3-4e9a-981a-49a892acc54c_isrForCrosstalkSources_1125922_13.stderr parsl.dataflow.dflow INFO: Task 3058 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b35204056f2ed76da5245eb4e6db12a2 with result from task 3058 parsl.dataflow.dflow DEBUG: Task 3058 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/57d0c219-0535-47ea-85b9-fe024c153abe_isrForCrosstalkSources_1125915_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/57d0c219-0535-47ea-85b9-fe024c153abe_isrForCrosstalkSources_1125915_33.stderr parsl.dataflow.dflow INFO: Task 3050 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9ee0178873de630d289d4853c42e615 with result from task 3050 parsl.dataflow.dflow DEBUG: Task 3050 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/12300f5b-5c69-4055-9403-20e0758f375c_isrForCrosstalkSources_1125910_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/12300f5b-5c69-4055-9403-20e0758f375c_isrForCrosstalkSources_1125910_56.stderr parsl.dataflow.dflow INFO: Task 3039 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60d7f27acd334d49150f5583f805c58a with result from task 3039 parsl.dataflow.dflow DEBUG: Task 3039 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/95921566-d44f-4edb-8844-c0c3fa4222ea_isrForCrosstalkSources_1125921_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/95921566-d44f-4edb-8844-c0c3fa4222ea_isrForCrosstalkSources_1125921_19.stderr parsl.dataflow.dflow INFO: Task 3086 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a40e31ba4597dea0fb9cda29d355afc with result from task 3086 parsl.dataflow.dflow DEBUG: Task 3086 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/dff9bda1-5fde-40b8-bbd7-26d5179fd45e_isrForCrosstalkSources_1125943_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/dff9bda1-5fde-40b8-bbd7-26d5179fd45e_isrForCrosstalkSources_1125943_17.stderr parsl.dataflow.dflow INFO: Task 3084 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 231b87498a9fe27a2e6ebd2afd6ab9aa with result from task 3084 parsl.dataflow.dflow DEBUG: Task 3084 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/b6dfd2bb-c62a-4bd0-9878-f12f92d2fba6_isrForCrosstalkSources_1125902_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/b6dfd2bb-c62a-4bd0-9878-f12f92d2fba6_isrForCrosstalkSources_1125902_51.stderr parsl.dataflow.dflow INFO: Task 2923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 574cf1fd6e35ca0b2e7fb81e3a5d52db with result from task 2923 parsl.dataflow.dflow DEBUG: Task 2923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/9fc38d0b-a1a6-4fb8-9744-45a97e4e1b04_isrForCrosstalkSources_1125930_4.stdout parsl.dataflow.dflow INFO: Standard error for task 2923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/9fc38d0b-a1a6-4fb8-9744-45a97e4e1b04_isrForCrosstalkSources_1125930_4.stderr parsl.dataflow.dflow INFO: Task 2998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3014a59957d3a9631e2a68e79e49a6e with result from task 2998 parsl.dataflow.dflow DEBUG: Task 2998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/8bbdf2e8-36c8-4a24-9f01-bb2d3ed9484b_isrForCrosstalkSources_1125915_14.stdout parsl.dataflow.dflow INFO: Standard error for task 2998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/8bbdf2e8-36c8-4a24-9f01-bb2d3ed9484b_isrForCrosstalkSources_1125915_14.stderr parsl.dataflow.dflow INFO: Task 3100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0126d25ab7464d4684f4d6f120ebd9c9 with result from task 3100 parsl.dataflow.dflow DEBUG: Task 3100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/527fa5e0-3b80-4a2f-a16a-4f752d1628e3_isrForCrosstalkSources_1125903_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/527fa5e0-3b80-4a2f-a16a-4f752d1628e3_isrForCrosstalkSources_1125903_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:18 proc_lsst.multi:146 [INFO] found job 16896500 in provider checkpoint proc_lsst.multi INFO: found job 16896500 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3052 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e0e3456ea9b279a41ed2b4d576466e9 with result from task 3052 parsl.dataflow.dflow DEBUG: Task 3052 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e911e6b5-5173-4fbe-9beb-e6c738287bed_isrForCrosstalkSources_1125929_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/e911e6b5-5173-4fbe-9beb-e6c738287bed_isrForCrosstalkSources_1125929_40.stderr parsl.dataflow.dflow INFO: Task 3037 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1167f2233555b16bb05282992a642d05 with result from task 3037 parsl.dataflow.dflow DEBUG: Task 3037 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/1f20602e-98cc-4a8f-ae35-05e1ee7a6564_isrForCrosstalkSources_1125916_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/1f20602e-98cc-4a8f-ae35-05e1ee7a6564_isrForCrosstalkSources_1125916_32.stderr parsl.dataflow.dflow INFO: Task 2823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a13c84ec6db46e9aebac7f0fbb0000c3 with result from task 2823 parsl.dataflow.dflow DEBUG: Task 2823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7af73753-c781-4db1-8d7f-c1cfffa74a70_isrForCrosstalkSources_1125929_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/7af73753-c781-4db1-8d7f-c1cfffa74a70_isrForCrosstalkSources_1125929_54.stderr parsl.dataflow.dflow INFO: Task 3047 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b9a3db3c2c96bb720fa435ad1282e8a with result from task 3047 parsl.dataflow.dflow DEBUG: Task 3047 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/f39ac24b-8358-4cbf-8202-8f6584bdd3ca_isrForCrosstalkSources_1125898_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/f39ac24b-8358-4cbf-8202-8f6584bdd3ca_isrForCrosstalkSources_1125898_40.stderr parsl.dataflow.dflow INFO: Task 2994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9b93629594a3d97b72ccd3868e81d89 with result from task 2994 parsl.dataflow.dflow DEBUG: Task 2994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/fb1cd9c7-21ff-47c2-9394-ba0284c01b32_isrForCrosstalkSources_1125893_6.stdout parsl.dataflow.dflow INFO: Standard error for task 2994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/fb1cd9c7-21ff-47c2-9394-ba0284c01b32_isrForCrosstalkSources_1125893_6.stderr parsl.dataflow.dflow INFO: Task 3036 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 749f850092998f67a1aa8ab38a7e2b33 with result from task 3036 parsl.dataflow.dflow DEBUG: Task 3036 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/e0c98b9e-9510-4d64-ba18-bc88cdc3ee89_isrForCrosstalkSources_1125897_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/e0c98b9e-9510-4d64-ba18-bc88cdc3ee89_isrForCrosstalkSources_1125897_28.stderr parsl.dataflow.dflow INFO: Task 2852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58e79ab67886c69e2a17e56ded08485a with result from task 2852 parsl.dataflow.dflow DEBUG: Task 2852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/8c76dc9b-38bf-4cad-ace2-a51718e572bf_isrForCrosstalkSources_1125906_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/8c76dc9b-38bf-4cad-ace2-a51718e572bf_isrForCrosstalkSources_1125906_12.stderr parsl.dataflow.dflow INFO: Task 3117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29701ae6ef1e8bebe0ac89f563819b4c with result from task 3117 parsl.dataflow.dflow DEBUG: Task 3117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/0866482d-5c6d-4e2f-8bac-1c97f187e91a_isrForCrosstalkSources_1125924_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/0866482d-5c6d-4e2f-8bac-1c97f187e91a_isrForCrosstalkSources_1125924_9.stderr parsl.dataflow.dflow INFO: Task 3057 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4146a15314ec6be60686ca3ce967506f with result from task 3057 parsl.dataflow.dflow DEBUG: Task 3057 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c97825d7-9422-43d7-ad13-e36943e7bb36_isrForCrosstalkSources_1125893_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c97825d7-9422-43d7-ad13-e36943e7bb36_isrForCrosstalkSources_1125893_34.stderr parsl.dataflow.dflow INFO: Task 2970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1da4cf9c2467300755f3635de0803e8e with result from task 2970 parsl.dataflow.dflow DEBUG: Task 2970 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/b46eb4e9-c53e-44be-9145-835b9588b339_isrForCrosstalkSources_1125918_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/b46eb4e9-c53e-44be-9145-835b9588b339_isrForCrosstalkSources_1125918_18.stderr parsl.dataflow.dflow INFO: Task 3034 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9b62d66c70ad5f6bb608454de5b406f with result from task 3034 parsl.dataflow.dflow DEBUG: Task 3034 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/98f88202-b26e-4c8a-b55a-162375bb156a_isrForCrosstalkSources_1125895_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/98f88202-b26e-4c8a-b55a-162375bb156a_isrForCrosstalkSources_1125895_50.stderr parsl.dataflow.dflow INFO: Task 3064 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7f62c09b3f1954b2ba782d956d3e92c with result from task 3064 parsl.dataflow.dflow DEBUG: Task 3064 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/21bab288-0fef-4476-a515-f15a473fd675_isrForCrosstalkSources_1125917_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/21bab288-0fef-4476-a515-f15a473fd675_isrForCrosstalkSources_1125917_12.stderr parsl.dataflow.dflow INFO: Task 3108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 622abd670b06e340b8b5c3281283e6ef with result from task 3108 parsl.dataflow.dflow DEBUG: Task 3108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/ce20d434-50e7-4c59-be78-41c0cc4b578c_isrForCrosstalkSources_1125932_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/ce20d434-50e7-4c59-be78-41c0cc4b578c_isrForCrosstalkSources_1125932_2.stderr parsl.dataflow.dflow INFO: Task 3071 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf4451a1755d65db0ed94a471638da85 with result from task 3071 parsl.dataflow.dflow DEBUG: Task 3071 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/950b802a-3bad-40f8-b699-e5f3976a3254_isrForCrosstalkSources_1125944_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/950b802a-3bad-40f8-b699-e5f3976a3254_isrForCrosstalkSources_1125944_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:19 proc_lsst.multi:146 [INFO] found job 16896501 in provider checkpoint proc_lsst.multi INFO: found job 16896501 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3029 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a0e79966ede4b6839539c165e6e1e87 with result from task 3029 parsl.dataflow.dflow DEBUG: Task 3029 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/24ad4f41-104a-4c26-93e5-f4e1d4e618a2_isrForCrosstalkSources_1125915_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/24ad4f41-104a-4c26-93e5-f4e1d4e618a2_isrForCrosstalkSources_1125915_12.stderr parsl.dataflow.dflow INFO: Task 3128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 778216ca49644b71a18ab68c9a3cc48f with result from task 3128 parsl.dataflow.dflow DEBUG: Task 3128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/02b70dfa-b0ac-4d62-a336-3909eec1efa0_isrForCrosstalkSources_1125915_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/02b70dfa-b0ac-4d62-a336-3909eec1efa0_isrForCrosstalkSources_1125915_26.stderr parsl.dataflow.dflow INFO: Task 2937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31e0795f57b37732b4753afb251a61b0 with result from task 2937 parsl.dataflow.dflow DEBUG: Task 2937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/41b12a68-18e7-4b66-9e50-284c44177dcd_isrForCrosstalkSources_1125928_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/41b12a68-18e7-4b66-9e50-284c44177dcd_isrForCrosstalkSources_1125928_42.stderr parsl.dataflow.dflow INFO: Task 2941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c02f1c5edce5846a4de9561e4ed1481c with result from task 2941 parsl.dataflow.dflow DEBUG: Task 2941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/6aae6415-e57a-4dc2-a467-35f5a58c8aeb_isrForCrosstalkSources_1125926_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/6aae6415-e57a-4dc2-a467-35f5a58c8aeb_isrForCrosstalkSources_1125926_28.stderr parsl.dataflow.dflow INFO: Task 3022 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a91c814d16b3eab440824e9280deb315 with result from task 3022 parsl.dataflow.dflow DEBUG: Task 3022 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d37d63ec-0af3-4bf7-a2f9-b33fc66519dd_isrForCrosstalkSources_1125915_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d37d63ec-0af3-4bf7-a2f9-b33fc66519dd_isrForCrosstalkSources_1125915_58.stderr parsl.dataflow.dflow INFO: Task 3116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7d012bce2b275d66610685c60d8355f with result from task 3116 parsl.dataflow.dflow DEBUG: Task 3116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/d1812e71-5bf5-4dc7-ae35-6ceaf6ca6c26_isrForCrosstalkSources_1125895_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/d1812e71-5bf5-4dc7-ae35-6ceaf6ca6c26_isrForCrosstalkSources_1125895_37.stderr parsl.dataflow.dflow INFO: Task 3118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 937960950fb8a094e738dc7568f7a343 with result from task 3118 parsl.dataflow.dflow DEBUG: Task 3118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/5ae12768-381b-40f4-ac84-1b8adc9ed2df_isrForCrosstalkSources_1125897_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/5ae12768-381b-40f4-ac84-1b8adc9ed2df_isrForCrosstalkSources_1125897_12.stderr parsl.dataflow.dflow INFO: Task 3041 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d145e0ce7b83e0bda0cebc3de934233f with result from task 3041 parsl.dataflow.dflow DEBUG: Task 3041 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/1f2d1dd2-1c47-4618-8772-dc94c99c6927_isrForCrosstalkSources_1125925_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/1f2d1dd2-1c47-4618-8772-dc94c99c6927_isrForCrosstalkSources_1125925_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:19 proc_lsst.multi:146 [INFO] found job 16896502 in provider checkpoint proc_lsst.multi INFO: found job 16896502 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3053 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a5d42d98ce93a32d238412a4a99b96d with result from task 3053 parsl.dataflow.dflow DEBUG: Task 3053 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/2fa624e9-6cb4-4bfa-8b9d-09aceebd3a3f_isrForCrosstalkSources_1125913_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/2fa624e9-6cb4-4bfa-8b9d-09aceebd3a3f_isrForCrosstalkSources_1125913_38.stderr parsl.dataflow.dflow INFO: Task 3051 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6be2333d620683f1868d9add4b6008d with result from task 3051 parsl.dataflow.dflow DEBUG: Task 3051 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/5f2947d9-2b44-4c26-bde9-a44490fbf8c7_isrForCrosstalkSources_1125917_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/5f2947d9-2b44-4c26-bde9-a44490fbf8c7_isrForCrosstalkSources_1125917_13.stderr parsl.dataflow.dflow INFO: Task 2877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 416d7223a62caba21733002e52ff3f11 with result from task 2877 parsl.dataflow.dflow DEBUG: Task 2877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/f97ad47c-2bb9-4016-a25c-313d00aeb79b_isrForCrosstalkSources_1125921_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/f97ad47c-2bb9-4016-a25c-313d00aeb79b_isrForCrosstalkSources_1125921_36.stderr parsl.dataflow.dflow INFO: Task 3066 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c77ff81e9e9d4c5a1a187a32bea35a3 with result from task 3066 parsl.dataflow.dflow DEBUG: Task 3066 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/0aeea548-55d1-4580-be88-120930268e2b_isrForCrosstalkSources_1125901_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/0aeea548-55d1-4580-be88-120930268e2b_isrForCrosstalkSources_1125901_3.stderr parsl.dataflow.dflow INFO: Task 3099 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46924675d2ffe5355a136010fd23e24f with result from task 3099 parsl.dataflow.dflow DEBUG: Task 3099 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/fb9d3df2-3b65-497c-995c-e2c92658ce4f_isrForCrosstalkSources_1125888_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/fb9d3df2-3b65-497c-995c-e2c92658ce4f_isrForCrosstalkSources_1125888_41.stderr parsl.dataflow.dflow INFO: Task 3045 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4facb071d425b839d5d595b2650122b with result from task 3045 parsl.dataflow.dflow DEBUG: Task 3045 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/aec9a431-5b71-4bd1-bc32-0e085561bb45_isrForCrosstalkSources_1125930_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/aec9a431-5b71-4bd1-bc32-0e085561bb45_isrForCrosstalkSources_1125930_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:19 proc_lsst.multi:146 [INFO] found job 16896503 in provider checkpoint proc_lsst.multi INFO: found job 16896503 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4096c9d5919f7393daa37e9889659322 with result from task 3012 parsl.dataflow.dflow DEBUG: Task 3012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/4156e10b-0984-4026-968b-b8093d03f874_isrForCrosstalkSources_1125892_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/4156e10b-0984-4026-968b-b8093d03f874_isrForCrosstalkSources_1125892_53.stderr parsl.dataflow.dflow INFO: Task 2892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3318064ff7292524e2a75436587c7f01 with result from task 2892 parsl.dataflow.dflow DEBUG: Task 2892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/8e7fdf66-a079-4e59-83a0-b2a763a845bc_isrForCrosstalkSources_1125933_54.stdout parsl.dataflow.dflow INFO: Standard error for task 2892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/8e7fdf66-a079-4e59-83a0-b2a763a845bc_isrForCrosstalkSources_1125933_54.stderr parsl.dataflow.dflow INFO: Task 2981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86e96f8715c7eb3173bc169496281e52 with result from task 2981 parsl.dataflow.dflow DEBUG: Task 2981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/5f0bb0d0-12ec-435f-8219-f763395ade6c_isrForCrosstalkSources_1125920_35.stdout parsl.dataflow.dflow INFO: Standard error for task 2981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/5f0bb0d0-12ec-435f-8219-f763395ade6c_isrForCrosstalkSources_1125920_35.stderr parsl.dataflow.dflow INFO: Task 3078 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38afaddc6ebe24211d82a8f15aee158f with result from task 3078 parsl.dataflow.dflow DEBUG: Task 3078 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/6dd377c8-d2dd-4b48-8d03-3a09e8167fee_isrForCrosstalkSources_1125936_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/6dd377c8-d2dd-4b48-8d03-3a09e8167fee_isrForCrosstalkSources_1125936_8.stderr parsl.dataflow.dflow INFO: Task 3030 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecb2283c108ea4e45c0f0d16e6cab66d with result from task 3030 parsl.dataflow.dflow DEBUG: Task 3030 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d1e787cd-9f68-4707-8784-2f0c9f46c170_isrForCrosstalkSources_1125911_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d1e787cd-9f68-4707-8784-2f0c9f46c170_isrForCrosstalkSources_1125911_43.stderr parsl.dataflow.dflow INFO: Task 3069 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d48ee7cab664638872c795df328741d with result from task 3069 parsl.dataflow.dflow DEBUG: Task 3069 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/40f665f9-5aed-4d14-b401-3c2aee56d183_isrForCrosstalkSources_1125918_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/40f665f9-5aed-4d14-b401-3c2aee56d183_isrForCrosstalkSources_1125918_13.stderr parsl.dataflow.dflow INFO: Task 3054 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6661e510ae6b916e88d369d9ed8f42cb with result from task 3054 parsl.dataflow.dflow DEBUG: Task 3054 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/85fce27e-56f4-4ad9-b449-a14dff4da330_isrForCrosstalkSources_1125913_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/85fce27e-56f4-4ad9-b449-a14dff4da330_isrForCrosstalkSources_1125913_35.stderr parsl.dataflow.dflow INFO: Task 2984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b313a60ae8423339efa55e7a182cbb20 with result from task 2984 parsl.dataflow.dflow DEBUG: Task 2984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/37fea12b-7f0d-4463-8362-cb8a01d9b658_isrForCrosstalkSources_1125892_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/37fea12b-7f0d-4463-8362-cb8a01d9b658_isrForCrosstalkSources_1125892_47.stderr parsl.dataflow.dflow INFO: Task 3007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cf0b47646a72f42734b67da41f73ec3 with result from task 3007 parsl.dataflow.dflow DEBUG: Task 3007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/f628086b-f665-4719-bd0b-22f03b3d60a4_isrForCrosstalkSources_1125904_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/f628086b-f665-4719-bd0b-22f03b3d60a4_isrForCrosstalkSources_1125904_61.stderr parsl.dataflow.dflow INFO: Task 2983 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c14029e844a453870e4826caf1704337 with result from task 2983 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:19 proc_lsst.multi:146 [INFO] found job 16896504 in provider checkpoint proc_lsst.multi INFO: found job 16896504 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2983 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/36991a94-b1c7-403a-9fe1-6e5ae5d3d79f_isrForCrosstalkSources_1125929_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/36991a94-b1c7-403a-9fe1-6e5ae5d3d79f_isrForCrosstalkSources_1125929_41.stderr parsl.dataflow.dflow INFO: Task 2955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbbb2583b6c13a2e92595a35fe070d93 with result from task 2955 parsl.dataflow.dflow DEBUG: Task 2955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/78e8f2e8-87f6-43fe-bf48-7fd3663e992e_isrForCrosstalkSources_1125930_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/78e8f2e8-87f6-43fe-bf48-7fd3663e992e_isrForCrosstalkSources_1125930_57.stderr parsl.dataflow.dflow INFO: Task 2928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39861f2baf6686c8483807e3aa49f554 with result from task 2928 parsl.dataflow.dflow DEBUG: Task 2928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/674c7253-d180-4261-90e0-4af7bda065d1_isrForCrosstalkSources_1125912_12.stdout parsl.dataflow.dflow INFO: Standard error for task 2928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/674c7253-d180-4261-90e0-4af7bda065d1_isrForCrosstalkSources_1125912_12.stderr parsl.dataflow.dflow INFO: Task 3000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 832dd59e870f09f5e616d4dcb7dd7509 with result from task 3000 parsl.dataflow.dflow DEBUG: Task 3000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/8e9eb5d0-2142-4dba-9f7a-6a52c8caaba0_isrForCrosstalkSources_1125939_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/8e9eb5d0-2142-4dba-9f7a-6a52c8caaba0_isrForCrosstalkSources_1125939_38.stderr parsl.dataflow.dflow INFO: Task 3092 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f57e4521ecc7deb05214e1c53d97b506 with result from task 3092 parsl.dataflow.dflow DEBUG: Task 3092 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/893029c4-329c-47f7-bcf5-7bf37becda87_isrForCrosstalkSources_1125917_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/893029c4-329c-47f7-bcf5-7bf37becda87_isrForCrosstalkSources_1125917_59.stderr parsl.dataflow.dflow INFO: Task 2991 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2123e63ec343e92ab71a4e8401881684 with result from task 2991 parsl.dataflow.dflow DEBUG: Task 2991 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/f2b6a4bd-70d5-4586-ac0d-973c3ac94d16_isrForCrosstalkSources_1125900_9.stdout parsl.dataflow.dflow INFO: Standard error for task 2991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/f2b6a4bd-70d5-4586-ac0d-973c3ac94d16_isrForCrosstalkSources_1125900_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:20 proc_lsst.multi:146 [INFO] found job 16896505 in provider checkpoint proc_lsst.multi INFO: found job 16896505 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 2927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 905f051ed16340e3017fbeb7f060069e with result from task 2927 parsl.dataflow.dflow DEBUG: Task 2927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f3698660-d36f-45d7-ba21-ff92b47309fa_isrForCrosstalkSources_1125945_10.stdout parsl.dataflow.dflow INFO: Standard error for task 2927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f3698660-d36f-45d7-ba21-ff92b47309fa_isrForCrosstalkSources_1125945_10.stderr parsl.dataflow.dflow INFO: Task 3040 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 673a7f0a13cc4b1714b9f853e37ce4a9 with result from task 3040 parsl.dataflow.dflow DEBUG: Task 3040 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/7201b000-c288-4a89-964c-e52872890580_isrForCrosstalkSources_1125909_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/7201b000-c288-4a89-964c-e52872890580_isrForCrosstalkSources_1125909_61.stderr parsl.dataflow.dflow INFO: Task 2872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4700e9847ee1f831ffe013855a517f0 with result from task 2872 parsl.dataflow.dflow DEBUG: Task 2872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/6e0d7666-85ab-4868-96a7-b5d461503522_isrForCrosstalkSources_1125915_52.stdout parsl.dataflow.dflow INFO: Standard error for task 2872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/6e0d7666-85ab-4868-96a7-b5d461503522_isrForCrosstalkSources_1125915_52.stderr parsl.dataflow.dflow INFO: Task 3015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edfe436d632d57483aa702b9f06a42e4 with result from task 3015 parsl.dataflow.dflow DEBUG: Task 3015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/b82e9498-79bd-45cc-9706-d16bf60f8b2f_isrForCrosstalkSources_1125916_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/b82e9498-79bd-45cc-9706-d16bf60f8b2f_isrForCrosstalkSources_1125916_18.stderr parsl.dataflow.dflow INFO: Task 2976 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c97456a6e01236528f4370bfc382ec3 with result from task 2976 parsl.dataflow.dflow DEBUG: Task 2976 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/ad78c7b0-6ce3-4e9f-b6fc-a10ac705a445_isrForCrosstalkSources_1125895_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/ad78c7b0-6ce3-4e9f-b6fc-a10ac705a445_isrForCrosstalkSources_1125895_36.stderr parsl.dataflow.dflow INFO: Task 3049 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07807e96d5bd990adff867945f799e46 with result from task 3049 parsl.dataflow.dflow DEBUG: Task 3049 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/82ebe9b7-c0de-45fc-a62c-340816580a08_isrForCrosstalkSources_1125890_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/82ebe9b7-c0de-45fc-a62c-340816580a08_isrForCrosstalkSources_1125890_48.stderr parsl.dataflow.dflow INFO: Task 3038 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 313caea5ef87808f1242c70be17c70d2 with result from task 3038 parsl.dataflow.dflow DEBUG: Task 3038 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/aa987415-458d-48f0-a822-34e4d6a5923a_isrForCrosstalkSources_1125898_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/aa987415-458d-48f0-a822-34e4d6a5923a_isrForCrosstalkSources_1125898_16.stderr parsl.dataflow.dflow INFO: Task 3077 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d5de52a6c288185d89b68ec37764eac with result from task 3077 parsl.dataflow.dflow DEBUG: Task 3077 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/49063da9-9778-46e4-b617-17f2c25c8316_isrForCrosstalkSources_1125919_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/49063da9-9778-46e4-b617-17f2c25c8316_isrForCrosstalkSources_1125919_42.stderr parsl.dataflow.dflow INFO: Task 3134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5b6e30da57650491235407553b8e0c0 with result from task 3134 parsl.dataflow.dflow DEBUG: Task 3134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/889fe82e-22fb-4258-a4a8-612900e7a7e0_isrForCrosstalkSources_1125926_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/889fe82e-22fb-4258-a4a8-612900e7a7e0_isrForCrosstalkSources_1125926_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:20 proc_lsst.multi:146 [INFO] found job 16896506 in provider checkpoint proc_lsst.multi INFO: found job 16896506 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3027 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ddbb7a773feac5b41ced2c22a917a0f with result from task 3027 parsl.dataflow.dflow DEBUG: Task 3027 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/02308bdb-75a7-434c-9f6c-6bfa2b521f70_isrForCrosstalkSources_1125918_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/02308bdb-75a7-434c-9f6c-6bfa2b521f70_isrForCrosstalkSources_1125918_52.stderr parsl.dataflow.dflow INFO: Task 3060 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c349b21b251e202e66baefc0be6e682 with result from task 3060 parsl.dataflow.dflow DEBUG: Task 3060 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/03962a24-16dc-4f72-9172-b7aaef2cbe96_isrForCrosstalkSources_1125924_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/03962a24-16dc-4f72-9172-b7aaef2cbe96_isrForCrosstalkSources_1125924_61.stderr parsl.dataflow.dflow INFO: Task 3114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8564ccd1fde57e6d9ba60fe813271f74 with result from task 3114 parsl.dataflow.dflow DEBUG: Task 3114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/db83d08f-324e-43b8-801d-758eef971c99_isrForCrosstalkSources_1125922_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/db83d08f-324e-43b8-801d-758eef971c99_isrForCrosstalkSources_1125922_2.stderr parsl.dataflow.dflow INFO: Task 3019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f62d03a096e872d87cfec204286f6f1 with result from task 3019 parsl.dataflow.dflow DEBUG: Task 3019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/9d0a2aa7-4d02-4fd7-9270-8d5c91d7fae8_isrForCrosstalkSources_1125888_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/9d0a2aa7-4d02-4fd7-9270-8d5c91d7fae8_isrForCrosstalkSources_1125888_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:20 proc_lsst.multi:146 [INFO] found job 16896507 in provider checkpoint proc_lsst.multi INFO: found job 16896507 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3073 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13fd20988c43a6d92b30f370c18335dd with result from task 3073 parsl.dataflow.dflow DEBUG: Task 3073 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/1044760c-22c4-4c80-8b89-98f9b265963c_isrForCrosstalkSources_1125941_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/1044760c-22c4-4c80-8b89-98f9b265963c_isrForCrosstalkSources_1125941_45.stderr parsl.dataflow.dflow INFO: Task 3080 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db47d44df2b065ecc24cbee095d85f4e with result from task 3080 parsl.dataflow.dflow DEBUG: Task 3080 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/cc32b263-8b16-4074-8205-74731d3bf67a_isrForCrosstalkSources_1125925_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/cc32b263-8b16-4074-8205-74731d3bf67a_isrForCrosstalkSources_1125925_37.stderr parsl.dataflow.dflow INFO: Task 3144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5e3dc84f3aabaa8da7e0e020874e0b8 with result from task 3144 parsl.dataflow.dflow DEBUG: Task 3144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/df683b0e-692e-4a43-86e8-c31a1d7e4884_isrForCrosstalkSources_1125918_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/df683b0e-692e-4a43-86e8-c31a1d7e4884_isrForCrosstalkSources_1125918_45.stderr parsl.dataflow.dflow INFO: Task 3093 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28fc0ec692d6b45a93cad62ccf3a0291 with result from task 3093 parsl.dataflow.dflow DEBUG: Task 3093 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/5d9e82e7-5c11-49c6-936c-502ed1830c1c_isrForCrosstalkSources_1125901_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/5d9e82e7-5c11-49c6-936c-502ed1830c1c_isrForCrosstalkSources_1125901_5.stderr parsl.dataflow.dflow INFO: Task 3096 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c898a273fcb1896f0257d2e060f41941 with result from task 3096 parsl.dataflow.dflow DEBUG: Task 3096 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/48a779a5-10c5-4e56-ba7c-e18e784de7a6_isrForCrosstalkSources_1125931_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/48a779a5-10c5-4e56-ba7c-e18e784de7a6_isrForCrosstalkSources_1125931_12.stderr parsl.dataflow.dflow INFO: Task 3079 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36c55e94f3fa4a2f8a019195890ecd15 with result from task 3079 parsl.dataflow.dflow DEBUG: Task 3079 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/dbb17ed1-e904-4cc3-8df7-b7b4885913fd_isrForCrosstalkSources_1125939_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/dbb17ed1-e904-4cc3-8df7-b7b4885913fd_isrForCrosstalkSources_1125939_10.stderr parsl.dataflow.dflow INFO: Task 2999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c990bbade57bf429c7952c167c23e1de with result from task 2999 parsl.dataflow.dflow DEBUG: Task 2999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/10f0567b-3e96-42a8-a725-f9226505844e_isrForCrosstalkSources_1125890_41.stdout parsl.dataflow.dflow INFO: Standard error for task 2999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/10f0567b-3e96-42a8-a725-f9226505844e_isrForCrosstalkSources_1125890_41.stderr parsl.dataflow.dflow INFO: Task 3140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5ef6c03903651998480123581267b28 with result from task 3140 parsl.dataflow.dflow DEBUG: Task 3140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/13b7d01b-189b-4636-b501-d50b6724291c_isrForCrosstalkSources_1125935_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/13b7d01b-189b-4636-b501-d50b6724291c_isrForCrosstalkSources_1125935_17.stderr parsl.dataflow.dflow INFO: Task 3044 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 710a15d4596fd4f3304489558ee74bca with result from task 3044 parsl.dataflow.dflow DEBUG: Task 3044 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/bb0230a7-fdf2-4371-87c2-4fe24caf7e44_isrForCrosstalkSources_1125928_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/bb0230a7-fdf2-4371-87c2-4fe24caf7e44_isrForCrosstalkSources_1125928_44.stderr parsl.dataflow.dflow INFO: Task 2948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a74c062c11f524a3c4458b06fe558fba with result from task 2948 parsl.dataflow.dflow DEBUG: Task 2948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/d44992c2-5e0a-4155-b0de-06a7edf90e51_isrForCrosstalkSources_1125929_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/d44992c2-5e0a-4155-b0de-06a7edf90e51_isrForCrosstalkSources_1125929_47.stderr parsl.dataflow.dflow INFO: Task 3137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bffb0e0147c6c5c0457cebe6156de26f with result from task 3137 parsl.dataflow.dflow DEBUG: Task 3137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f7aae486-a69f-4184-9d42-3e9506416e65_isrForCrosstalkSources_1125939_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f7aae486-a69f-4184-9d42-3e9506416e65_isrForCrosstalkSources_1125939_6.stderr parsl.dataflow.dflow INFO: Task 2788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df1febd32ac887260085cf548319a759 with result from task 2788 parsl.dataflow.dflow DEBUG: Task 2788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/40c31a75-8013-4319-9051-e67b0058fefb_isrForCrosstalkSources_1125903_39.stdout parsl.dataflow.dflow INFO: Standard error for task 2788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/40c31a75-8013-4319-9051-e67b0058fefb_isrForCrosstalkSources_1125903_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:21 proc_lsst.multi:146 [INFO] found job 16896508 in provider checkpoint proc_lsst.multi INFO: found job 16896508 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad3578d71d359b08506b50c82cad9a85 with result from task 3120 parsl.dataflow.dflow DEBUG: Task 3120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/9bc06530-61e2-4d32-ba7c-e33cf41270e6_isrForCrosstalkSources_1125900_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/9bc06530-61e2-4d32-ba7c-e33cf41270e6_isrForCrosstalkSources_1125900_58.stderr parsl.dataflow.dflow INFO: Task 3005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2488ae56e3d6133ff608a7acc495f107 with result from task 3005 parsl.dataflow.dflow DEBUG: Task 3005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/e7490eec-72de-4667-887b-ecb66001a62d_isrForCrosstalkSources_1125898_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/e7490eec-72de-4667-887b-ecb66001a62d_isrForCrosstalkSources_1125898_54.stderr parsl.dataflow.dflow INFO: Task 3095 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b828ed01fe61b8223efdab8aa61e740 with result from task 3095 parsl.dataflow.dflow DEBUG: Task 3095 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/7ea372c0-506f-4b56-8d09-4276fcae9f94_isrForCrosstalkSources_1125906_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/7ea372c0-506f-4b56-8d09-4276fcae9f94_isrForCrosstalkSources_1125906_43.stderr parsl.dataflow.dflow INFO: Task 3033 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9eb054b9dd2ad5c63acaa9b2848e9c6 with result from task 3033 parsl.dataflow.dflow DEBUG: Task 3033 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/950325fa-80f8-4e4c-9e6d-a581771e2758_isrForCrosstalkSources_1125925_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/950325fa-80f8-4e4c-9e6d-a581771e2758_isrForCrosstalkSources_1125925_50.stderr parsl.dataflow.dflow INFO: Task 3126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 393ac875c1ce491f4b31ee4263648f5d with result from task 3126 parsl.dataflow.dflow DEBUG: Task 3126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/0b8b7046-476e-486f-88c1-d095642c5734_isrForCrosstalkSources_1125915_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/0b8b7046-476e-486f-88c1-d095642c5734_isrForCrosstalkSources_1125915_56.stderr parsl.dataflow.dflow INFO: Task 3094 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c743cad559fed6db3af13144dc559ce with result from task 3094 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:21 proc_lsst.multi:146 [INFO] found job 16896509 in provider checkpoint proc_lsst.multi INFO: found job 16896509 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 3094 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/d1d21181-9921-4a51-8e33-d2cacd80a7a6_isrForCrosstalkSources_1125924_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/d1d21181-9921-4a51-8e33-d2cacd80a7a6_isrForCrosstalkSources_1125924_31.stderr parsl.dataflow.dflow INFO: Task 3003 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 384634b3a823a7fbd5aa7537376f6ab0 with result from task 3003 parsl.dataflow.dflow DEBUG: Task 3003 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1c054595-3852-4bba-b4cb-024047fcfbc2_isrForCrosstalkSources_1125900_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/1c054595-3852-4bba-b4cb-024047fcfbc2_isrForCrosstalkSources_1125900_36.stderr parsl.dataflow.dflow INFO: Task 3129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc76648146f42e9744b41d7db1325c02 with result from task 3129 parsl.dataflow.dflow DEBUG: Task 3129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/b157fcfd-a27a-4196-9b5d-8ae66a95edd5_isrForCrosstalkSources_1125916_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/b157fcfd-a27a-4196-9b5d-8ae66a95edd5_isrForCrosstalkSources_1125916_49.stderr parsl.dataflow.dflow INFO: Task 2947 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1808fbad27dcb952e6282ca9ec257e75 with result from task 2947 parsl.dataflow.dflow DEBUG: Task 2947 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/34d1c081-8007-47d1-9762-2a4ab7e01a92_isrForCrosstalkSources_1125939_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/34d1c081-8007-47d1-9762-2a4ab7e01a92_isrForCrosstalkSources_1125939_29.stderr parsl.dataflow.dflow INFO: Task 3065 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8000ffab4df8ae91242bcb0450ae2fe8 with result from task 3065 parsl.dataflow.dflow DEBUG: Task 3065 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/5b35b883-8d34-4a9b-af91-a1380fa7e53f_isrForCrosstalkSources_1125894_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/5b35b883-8d34-4a9b-af91-a1380fa7e53f_isrForCrosstalkSources_1125894_43.stderr parsl.dataflow.dflow INFO: Task 3133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2dcba3814e836cbf9e9c176270606f1 with result from task 3133 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:22 proc_lsst.multi:146 [INFO] found job 16896510 in provider checkpoint proc_lsst.multi INFO: found job 16896510 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 3133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/5c5d4026-0108-40eb-8e81-81e0ceeb5352_isrForCrosstalkSources_1125945_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/5c5d4026-0108-40eb-8e81-81e0ceeb5352_isrForCrosstalkSources_1125945_59.stderr parsl.dataflow.dflow INFO: Task 3157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ec97b829695e8b85c6de2b599fc794b with result from task 3157 parsl.dataflow.dflow DEBUG: Task 3157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/512764d4-ba7d-4547-8679-4e726d9d30a3_isrForCrosstalkSources_1125896_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/512764d4-ba7d-4547-8679-4e726d9d30a3_isrForCrosstalkSources_1125896_32.stderr parsl.dataflow.dflow INFO: Task 3081 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54b587df18f691953c874bd8b15ce185 with result from task 3081 parsl.dataflow.dflow DEBUG: Task 3081 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/446332c1-991e-497d-9cb3-7fd78d0b2e57_isrForCrosstalkSources_1125940_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/446332c1-991e-497d-9cb3-7fd78d0b2e57_isrForCrosstalkSources_1125940_48.stderr parsl.dataflow.dflow INFO: Task 3043 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23a3e618de7fb0c92c7fd150933ca8a1 with result from task 3043 parsl.dataflow.dflow DEBUG: Task 3043 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9d450f53-dc93-4f8f-8898-0b02aa291d55_isrForCrosstalkSources_1125921_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/9d450f53-dc93-4f8f-8898-0b02aa291d55_isrForCrosstalkSources_1125921_53.stderr parsl.dataflow.dflow INFO: Task 3008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84ef20897f5419a7c3e041ba3d77f6c9 with result from task 3008 parsl.dataflow.dflow DEBUG: Task 3008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/d1119e89-3f11-48de-9140-459dedfb128b_isrForCrosstalkSources_1125909_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/d1119e89-3f11-48de-9140-459dedfb128b_isrForCrosstalkSources_1125909_50.stderr parsl.dataflow.dflow INFO: Task 3089 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82c8eb7bb0c72ab43ebf029b92ccf042 with result from task 3089 parsl.dataflow.dflow DEBUG: Task 3089 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/4da0ff6f-ab07-4b2e-8a31-1aea1738c0dc_isrForCrosstalkSources_1125902_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/4da0ff6f-ab07-4b2e-8a31-1aea1738c0dc_isrForCrosstalkSources_1125902_49.stderr parsl.dataflow.dflow INFO: Task 3028 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41819b0e1f55d75290692e8386ba3d1b with result from task 3028 parsl.dataflow.dflow DEBUG: Task 3028 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/ebcb53d4-6052-4411-b9f9-e4c08bba7911_isrForCrosstalkSources_1125943_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/ebcb53d4-6052-4411-b9f9-e4c08bba7911_isrForCrosstalkSources_1125943_5.stderr parsl.dataflow.dflow INFO: Task 3085 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 498bc7587dbf23a4ee1fca53624e8e5e with result from task 3085 parsl.dataflow.dflow DEBUG: Task 3085 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/ddf8da26-68ea-4ebc-9993-c4419839443c_isrForCrosstalkSources_1125941_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/ddf8da26-68ea-4ebc-9993-c4419839443c_isrForCrosstalkSources_1125941_57.stderr parsl.dataflow.dflow INFO: Task 3017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b30fe3960f0059ac5adfbd8fbe9a23f2 with result from task 3017 parsl.dataflow.dflow DEBUG: Task 3017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/cbf19a4d-2f2d-4236-ab69-e4050cefc7dc_isrForCrosstalkSources_1125896_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/cbf19a4d-2f2d-4236-ab69-e4050cefc7dc_isrForCrosstalkSources_1125896_33.stderr parsl.dataflow.dflow INFO: Task 2987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f80ed3b1ce5a2e3594c6b0cbf2f38a2 with result from task 2987 parsl.dataflow.dflow DEBUG: Task 2987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/2b16b540-ee5c-4d3a-b11d-cd6c8c4d4fa5_isrForCrosstalkSources_1125920_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/2b16b540-ee5c-4d3a-b11d-cd6c8c4d4fa5_isrForCrosstalkSources_1125920_34.stderr parsl.dataflow.dflow INFO: Task 2933 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0f5384300578211b791758a272c8720 with result from task 2933 parsl.dataflow.dflow DEBUG: Task 2933 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/6169340f-3d13-48a1-9246-d7992e1253ea_isrForCrosstalkSources_1125907_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/6169340f-3d13-48a1-9246-d7992e1253ea_isrForCrosstalkSources_1125907_48.stderr parsl.dataflow.dflow INFO: Task 3091 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29264b75c5e696f1f2b34f0329f8a66d with result from task 3091 parsl.dataflow.dflow DEBUG: Task 3091 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/6ccb47d8-0ea5-4208-9984-85b9a358aa04_isrForCrosstalkSources_1125946_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/6ccb47d8-0ea5-4208-9984-85b9a358aa04_isrForCrosstalkSources_1125946_42.stderr parsl.dataflow.dflow INFO: Task 3119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7db97415d69baced8a037b4403d3ffd with result from task 3119 parsl.dataflow.dflow DEBUG: Task 3119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/632e83ed-c15c-42c6-8e53-30aef12f2451_isrForCrosstalkSources_1125911_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/632e83ed-c15c-42c6-8e53-30aef12f2451_isrForCrosstalkSources_1125911_22.stderr parsl.dataflow.dflow INFO: Task 3122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95519ab8db2403d1c822736742fd9164 with result from task 3122 parsl.dataflow.dflow DEBUG: Task 3122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/f9f70594-2e10-40db-b8c1-594d890f5388_isrForCrosstalkSources_1125937_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/f9f70594-2e10-40db-b8c1-594d890f5388_isrForCrosstalkSources_1125937_14.stderr parsl.dataflow.dflow INFO: Task 2996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efa566550946689e331a45c303eabe05 with result from task 2996 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:22 proc_lsst.multi:146 [INFO] found job 16896511 in provider checkpoint proc_lsst.multi INFO: found job 16896511 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/fef92aa7-a9a9-4747-bc16-160b66f39f12_isrForCrosstalkSources_1125900_21.stdout parsl.dataflow.dflow INFO: Standard error for task 2996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/fef92aa7-a9a9-4747-bc16-160b66f39f12_isrForCrosstalkSources_1125900_21.stderr parsl.dataflow.dflow INFO: Task 3106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9b7987d6c65b8db5b301df7cfa2c073 with result from task 3106 parsl.dataflow.dflow DEBUG: Task 3106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/f26d2f3d-5610-472a-b05e-b458ed4740f2_isrForCrosstalkSources_1125947_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/f26d2f3d-5610-472a-b05e-b458ed4740f2_isrForCrosstalkSources_1125947_19.stderr parsl.dataflow.dflow INFO: Task 3023 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 929a0c4903cca8d1ba883067aae2a11b with result from task 3023 parsl.dataflow.dflow DEBUG: Task 3023 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/005411cc-84f0-48ea-9008-e428c9fb5278_isrForCrosstalkSources_1125891_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/005411cc-84f0-48ea-9008-e428c9fb5278_isrForCrosstalkSources_1125891_2.stderr parsl.dataflow.dflow INFO: Task 3130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9c20b0f994e7fddc5c86711d4daf09d with result from task 3130 parsl.dataflow.dflow DEBUG: Task 3130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/0758c20f-15a6-4d49-b5ce-08083339506e_isrForCrosstalkSources_1125931_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/0758c20f-15a6-4d49-b5ce-08083339506e_isrForCrosstalkSources_1125931_29.stderr parsl.dataflow.dflow INFO: Task 2816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 894364e1e6b595100913be153c5a453b with result from task 2816 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:23 proc_lsst.multi:146 [INFO] found job 16896519 in provider checkpoint proc_lsst.multi INFO: found job 16896519 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 2816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/8dbf2f07-d29f-451b-bcbf-c8fbbdbbcf10_isrForCrosstalkSources_1125921_56.stdout parsl.dataflow.dflow INFO: Standard error for task 2816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/8dbf2f07-d29f-451b-bcbf-c8fbbdbbcf10_isrForCrosstalkSources_1125921_56.stderr parsl.dataflow.dflow INFO: Task 2960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2529f2348348fd7e569b14c97071573a with result from task 2960 parsl.dataflow.dflow DEBUG: Task 2960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/87ec1d2a-6c39-4c64-bce3-a5b5f9aaf601_isrForCrosstalkSources_1125935_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/87ec1d2a-6c39-4c64-bce3-a5b5f9aaf601_isrForCrosstalkSources_1125935_59.stderr parsl.dataflow.dflow INFO: Task 3110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61be17bc2dd3bf26a6e32f14c6279fbd with result from task 3110 parsl.dataflow.dflow DEBUG: Task 3110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/38be1731-f94d-4cf0-9aae-cf5599c232fc_isrForCrosstalkSources_1125909_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/38be1731-f94d-4cf0-9aae-cf5599c232fc_isrForCrosstalkSources_1125909_44.stderr parsl.dataflow.dflow INFO: Task 2810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68d3acada69a71e6102ca41fa2995b25 with result from task 2810 parsl.dataflow.dflow DEBUG: Task 2810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/23ac63f4-a4a1-4add-a71b-e16519afe112_isrForCrosstalkSources_1125931_48.stdout parsl.dataflow.dflow INFO: Standard error for task 2810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/23ac63f4-a4a1-4add-a71b-e16519afe112_isrForCrosstalkSources_1125931_48.stderr parsl.dataflow.dflow INFO: Task 3035 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c96bc06ad7ac4c26abceeb59097eb2b2 with result from task 3035 parsl.dataflow.dflow DEBUG: Task 3035 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/9b64d7ce-3bd0-467f-8e45-c271f9af25c1_isrForCrosstalkSources_1125926_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/9b64d7ce-3bd0-467f-8e45-c271f9af25c1_isrForCrosstalkSources_1125926_36.stderr parsl.dataflow.dflow INFO: Task 3172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c02f972eaf45a2059970bb8c0548c10c with result from task 3172 parsl.dataflow.dflow DEBUG: Task 3172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/9d3db9d0-b9b3-4f2a-bd07-344547ab2191_isrForCrosstalkSources_1125941_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/9d3db9d0-b9b3-4f2a-bd07-344547ab2191_isrForCrosstalkSources_1125941_22.stderr parsl.dataflow.dflow INFO: Task 3062 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f566239fb2f1cae09ba5def063986e7d with result from task 3062 parsl.dataflow.dflow DEBUG: Task 3062 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/8460b97c-1a96-4512-879f-fc2629907178_isrForCrosstalkSources_1125924_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/8460b97c-1a96-4512-879f-fc2629907178_isrForCrosstalkSources_1125924_47.stderr parsl.dataflow.dflow INFO: Task 2962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2972095ecdca9fe50d2a5847ba1abd98 with result from task 2962 parsl.dataflow.dflow DEBUG: Task 2962 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/7a348888-7527-4810-a04f-c93c45ee78aa_isrForCrosstalkSources_1125936_31.stdout parsl.dataflow.dflow INFO: Standard error for task 2962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/7a348888-7527-4810-a04f-c93c45ee78aa_isrForCrosstalkSources_1125936_31.stderr parsl.dataflow.dflow INFO: Task 2974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8aded4ffeea63a7d5454c3561e3667d7 with result from task 2974 parsl.dataflow.dflow DEBUG: Task 2974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/bd1929f9-a941-48b2-b69e-9fca18b433a7_isrForCrosstalkSources_1125906_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/bd1929f9-a941-48b2-b69e-9fca18b433a7_isrForCrosstalkSources_1125906_42.stderr parsl.dataflow.dflow INFO: Task 3153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fcb90e30588039c082f5a55414e50d4 with result from task 3153 parsl.dataflow.dflow DEBUG: Task 3153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/8c83a99d-1dcf-494a-924b-eddd6ea65078_isrForCrosstalkSources_1125902_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/8c83a99d-1dcf-494a-924b-eddd6ea65078_isrForCrosstalkSources_1125902_23.stderr parsl.dataflow.dflow INFO: Task 3132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b29061f225f36fed72f90c851b27a39 with result from task 3132 parsl.dataflow.dflow DEBUG: Task 3132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c946a156-a327-42a4-801b-a8e3b87e656b_isrForCrosstalkSources_1125916_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c946a156-a327-42a4-801b-a8e3b87e656b_isrForCrosstalkSources_1125916_34.stderr parsl.dataflow.dflow INFO: Task 3136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a76f3943dd35c5ce1ee4ad8216423c6b with result from task 3136 parsl.dataflow.dflow DEBUG: Task 3136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a2f740b6-6775-43b3-8163-c0c426237d05_isrForCrosstalkSources_1125945_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a2f740b6-6775-43b3-8163-c0c426237d05_isrForCrosstalkSources_1125945_39.stderr parsl.dataflow.dflow INFO: Task 3090 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 227508cd19086f6b0a713b59fc9dbec1 with result from task 3090 parsl.dataflow.dflow DEBUG: Task 3090 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/02120c32-ed15-4771-ab1a-e8ad2fc1ded7_isrForCrosstalkSources_1125938_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/02120c32-ed15-4771-ab1a-e8ad2fc1ded7_isrForCrosstalkSources_1125938_32.stderr parsl.dataflow.dflow INFO: Task 2990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 957c38cd1460e62e239e0894b89bade6 with result from task 2990 parsl.dataflow.dflow DEBUG: Task 2990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/39b39e23-4b1f-480a-a550-480a189a7923_isrForCrosstalkSources_1125925_59.stdout parsl.dataflow.dflow INFO: Standard error for task 2990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/39b39e23-4b1f-480a-a550-480a189a7923_isrForCrosstalkSources_1125925_59.stderr parsl.dataflow.dflow INFO: Task 3151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2fec6e3f94973e66a7fcf5362e66313 with result from task 3151 parsl.dataflow.dflow DEBUG: Task 3151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2f250e38-f76e-43e0-b11f-1a68e3c8dbdb_isrForCrosstalkSources_1125898_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2f250e38-f76e-43e0-b11f-1a68e3c8dbdb_isrForCrosstalkSources_1125898_28.stderr parsl.dataflow.dflow INFO: Task 3105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28c4c479c011dd4dcb6aeb16cb159835 with result from task 3105 parsl.dataflow.dflow DEBUG: Task 3105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/80296b39-c113-4fbc-a833-4f9773557c80_isrForCrosstalkSources_1125914_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/80296b39-c113-4fbc-a833-4f9773557c80_isrForCrosstalkSources_1125914_44.stderr parsl.dataflow.dflow INFO: Task 3138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a416477e7c904c773b0a09e2530c0622 with result from task 3138 parsl.dataflow.dflow DEBUG: Task 3138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/78821392-951b-4d17-9ed7-fea41f6931f8_isrForCrosstalkSources_1125926_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/78821392-951b-4d17-9ed7-fea41f6931f8_isrForCrosstalkSources_1125926_29.stderr parsl.dataflow.dflow INFO: Task 2975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da76d45ac9294c89de2ef371b9875c46 with result from task 2975 parsl.dataflow.dflow DEBUG: Task 2975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/29066f96-219f-4e54-9087-6399f77c86be_isrForCrosstalkSources_1125931_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/29066f96-219f-4e54-9087-6399f77c86be_isrForCrosstalkSources_1125931_57.stderr parsl.dataflow.dflow INFO: Task 2972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1aae5d46dfdc8a0d6dda4e70c706c8d6 with result from task 2972 parsl.dataflow.dflow DEBUG: Task 2972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/0661384d-fb2f-407b-ba94-c9dbd5e59588_isrForCrosstalkSources_1125934_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/0661384d-fb2f-407b-ba94-c9dbd5e59588_isrForCrosstalkSources_1125934_62.stderr parsl.dataflow.dflow INFO: Task 3152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e562046357e341c5e4eb37dfa1e65551 with result from task 3152 parsl.dataflow.dflow DEBUG: Task 3152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/7983e4c5-15ee-4a6f-aae7-dd8dab896752_isrForCrosstalkSources_1125893_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/7983e4c5-15ee-4a6f-aae7-dd8dab896752_isrForCrosstalkSources_1125893_25.stderr parsl.dataflow.dflow INFO: Task 3158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 925918ee39c769d9dbcdec5fc4501acd with result from task 3158 parsl.dataflow.dflow DEBUG: Task 3158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/94a0eeae-89ea-41bc-be78-ec3e45b88ce9_isrForCrosstalkSources_1125939_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/94a0eeae-89ea-41bc-be78-ec3e45b88ce9_isrForCrosstalkSources_1125939_20.stderr parsl.dataflow.dflow INFO: Task 3115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26375d38ca2780a860996b38144258d4 with result from task 3115 parsl.dataflow.dflow DEBUG: Task 3115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/b5ef6a17-7fb6-4db5-a154-0d3030956cb6_isrForCrosstalkSources_1125897_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/b5ef6a17-7fb6-4db5-a154-0d3030956cb6_isrForCrosstalkSources_1125897_43.stderr parsl.dataflow.dflow INFO: Task 3161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba28b9f55db922dd338e2cd286cf96c6 with result from task 3161 parsl.dataflow.dflow DEBUG: Task 3161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/cfa5d851-1d2f-4aaa-ba76-8a74c672c572_isrForCrosstalkSources_1125935_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/cfa5d851-1d2f-4aaa-ba76-8a74c672c572_isrForCrosstalkSources_1125935_40.stderr parsl.dataflow.dflow INFO: Task 3059 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca62f50bbb50b6878bc4103741fbc5ea with result from task 3059 parsl.dataflow.dflow DEBUG: Task 3059 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/4ae9b81d-9a26-4f8f-b93c-e8f62c2bfd9a_isrForCrosstalkSources_1125932_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/4ae9b81d-9a26-4f8f-b93c-e8f62c2bfd9a_isrForCrosstalkSources_1125932_31.stderr parsl.dataflow.dflow INFO: Task 3006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5718760da7345ba81e86533d4fd4c26c with result from task 3006 parsl.dataflow.dflow DEBUG: Task 3006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/43073ba5-c24f-4bdf-8e07-99edcdc7faa0_isrForCrosstalkSources_1125896_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/43073ba5-c24f-4bdf-8e07-99edcdc7faa0_isrForCrosstalkSources_1125896_43.stderr parsl.dataflow.dflow INFO: Task 3131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85be6b8efe0c3a3610e7e0357dffb02d with result from task 3131 parsl.dataflow.dflow DEBUG: Task 3131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/d4418d20-653b-48d4-b7e6-3fcd468433e5_isrForCrosstalkSources_1125913_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/d4418d20-653b-48d4-b7e6-3fcd468433e5_isrForCrosstalkSources_1125913_41.stderr parsl.dataflow.dflow INFO: Task 3121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b25a638d59e7e5b24944aeee086e4b6f with result from task 3121 parsl.dataflow.dflow DEBUG: Task 3121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/a1831642-bf3a-4716-a166-9a35ecb841c6_isrForCrosstalkSources_1125944_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/a1831642-bf3a-4716-a166-9a35ecb841c6_isrForCrosstalkSources_1125944_42.stderr parsl.dataflow.dflow INFO: Task 3168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b89046d9b6ae77ffde112047edbaad3d with result from task 3168 parsl.dataflow.dflow DEBUG: Task 3168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/aa4c5be3-7bab-41fb-ade0-270f29312b00_isrForCrosstalkSources_1125926_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/aa4c5be3-7bab-41fb-ade0-270f29312b00_isrForCrosstalkSources_1125926_30.stderr parsl.dataflow.dflow INFO: Task 3125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42f0721c9b35634c08ee477f258d0d50 with result from task 3125 parsl.dataflow.dflow DEBUG: Task 3125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/c48efe97-7fb0-48ba-8e28-da84027fb4ab_isrForCrosstalkSources_1125939_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/c48efe97-7fb0-48ba-8e28-da84027fb4ab_isrForCrosstalkSources_1125939_62.stderr parsl.dataflow.dflow INFO: Task 3197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f337d777b700c909afdc70b5d28448a with result from task 3197 parsl.dataflow.dflow DEBUG: Task 3197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/88a4b89c-adb3-4bf2-bf7a-74f8fd11bb8c_isrForCrosstalkSources_1125944_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/88a4b89c-adb3-4bf2-bf7a-74f8fd11bb8c_isrForCrosstalkSources_1125944_13.stderr parsl.dataflow.dflow INFO: Task 3188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44c96ba91b4315a3ee37e9bca5aa5313 with result from task 3188 parsl.dataflow.dflow DEBUG: Task 3188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/055a1d81-8f3c-4a3c-a4fe-e6599727a94a_isrForCrosstalkSources_1125929_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/055a1d81-8f3c-4a3c-a4fe-e6599727a94a_isrForCrosstalkSources_1125929_38.stderr parsl.dataflow.dflow INFO: Task 3097 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d256e9e919d6ad16429ed4b6c53add03 with result from task 3097 parsl.dataflow.dflow DEBUG: Task 3097 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/4f857cf6-50cc-4b0d-a02c-0a38473481ee_isrForCrosstalkSources_1125947_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/4f857cf6-50cc-4b0d-a02c-0a38473481ee_isrForCrosstalkSources_1125947_44.stderr parsl.dataflow.dflow INFO: Task 3146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65f5d488b17cff4cbe0384fcba71dbb1 with result from task 3146 parsl.dataflow.dflow DEBUG: Task 3146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/37ea9595-cda6-481e-8f4c-c01aead239db_isrForCrosstalkSources_1125892_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/37ea9595-cda6-481e-8f4c-c01aead239db_isrForCrosstalkSources_1125892_23.stderr parsl.dataflow.dflow INFO: Task 3061 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 219dc45f62ad8a6941e3708ad6d80775 with result from task 3061 parsl.dataflow.dflow DEBUG: Task 3061 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2bcb105a-e793-48c0-8cdc-ad40e73c141c_isrForCrosstalkSources_1125898_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/2bcb105a-e793-48c0-8cdc-ad40e73c141c_isrForCrosstalkSources_1125898_27.stderr parsl.dataflow.dflow INFO: Task 3171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98e00efdb3c4370ab8a7dfc82896df0a with result from task 3171 parsl.dataflow.dflow DEBUG: Task 3171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/adcb5d9e-0da4-454a-b5ca-1b13a0a1908b_isrForCrosstalkSources_1125938_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/adcb5d9e-0da4-454a-b5ca-1b13a0a1908b_isrForCrosstalkSources_1125938_28.stderr parsl.dataflow.dflow INFO: Task 3101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 740c453a9441303468b0e18f73a1568b with result from task 3101 parsl.dataflow.dflow DEBUG: Task 3101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/1dfa2744-1cfd-4fd2-a666-86ed0ca7dbdf_isrForCrosstalkSources_1125918_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/1dfa2744-1cfd-4fd2-a666-86ed0ca7dbdf_isrForCrosstalkSources_1125918_44.stderr parsl.dataflow.dflow INFO: Task 3111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9902de824462247298a455e4a44fbd73 with result from task 3111 parsl.dataflow.dflow DEBUG: Task 3111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/895be516-ed03-4331-a6c9-70ee0b8de0e1_isrForCrosstalkSources_1125941_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/895be516-ed03-4331-a6c9-70ee0b8de0e1_isrForCrosstalkSources_1125941_60.stderr parsl.dataflow.dflow INFO: Task 3011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48fe04cfe280c62ed2d40c66658eabcd with result from task 3011 parsl.dataflow.dflow DEBUG: Task 3011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d48b3025-9f2f-45ac-b1f9-bcd726c3cc1e_isrForCrosstalkSources_1125911_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d48b3025-9f2f-45ac-b1f9-bcd726c3cc1e_isrForCrosstalkSources_1125911_47.stderr parsl.dataflow.dflow INFO: Task 3165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5149952dce08900cda9706da8df8f7b2 with result from task 3165 parsl.dataflow.dflow DEBUG: Task 3165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/6e1b1fe0-c8de-4830-b2af-af0cd9c17965_isrForCrosstalkSources_1125924_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/6e1b1fe0-c8de-4830-b2af-af0cd9c17965_isrForCrosstalkSources_1125924_34.stderr parsl.dataflow.dflow INFO: Task 3102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92f1e6298a914530f5951ce4686bc817 with result from task 3102 parsl.dataflow.dflow DEBUG: Task 3102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/be987c70-4555-4749-8c1d-dcff94f20002_isrForCrosstalkSources_1125905_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/be987c70-4555-4749-8c1d-dcff94f20002_isrForCrosstalkSources_1125905_49.stderr parsl.dataflow.dflow INFO: Task 3159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b075b41d6c2503c12650eb3795c62dc with result from task 3159 parsl.dataflow.dflow DEBUG: Task 3159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/aae7912e-5bdd-4564-ba27-70f6882b8fa9_isrForCrosstalkSources_1125925_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/aae7912e-5bdd-4564-ba27-70f6882b8fa9_isrForCrosstalkSources_1125925_20.stderr parsl.dataflow.dflow INFO: Task 3098 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12e93a0ce7fec327c6450b50f1f2e14f with result from task 3098 parsl.dataflow.dflow DEBUG: Task 3098 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/a961ecf8-8c8e-4d81-9c72-8ab5f196ce36_isrForCrosstalkSources_1125897_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/a961ecf8-8c8e-4d81-9c72-8ab5f196ce36_isrForCrosstalkSources_1125897_49.stderr parsl.dataflow.dflow INFO: Task 3147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d91232f82f68c2a7f429a742953e840d with result from task 3147 parsl.dataflow.dflow DEBUG: Task 3147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/f0e2ce19-5beb-48ac-bef8-01eb9c905540_isrForCrosstalkSources_1125931_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/f0e2ce19-5beb-48ac-bef8-01eb9c905540_isrForCrosstalkSources_1125931_44.stderr parsl.dataflow.dflow INFO: Task 2848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4170e9e9cc618a3be1cd9daa88b0e86 with result from task 2848 parsl.dataflow.dflow DEBUG: Task 2848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/52569885-3b08-4d50-ad0b-a0c5e691adb1_isrForCrosstalkSources_1125929_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/52569885-3b08-4d50-ad0b-a0c5e691adb1_isrForCrosstalkSources_1125929_36.stderr parsl.dataflow.dflow INFO: Task 3166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a165a74944a0458106c54b0ff211a5d with result from task 3166 parsl.dataflow.dflow DEBUG: Task 3166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/0c8a54cb-dc41-401f-a66f-0545bc8a7a06_isrForCrosstalkSources_1125936_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/0c8a54cb-dc41-401f-a66f-0545bc8a7a06_isrForCrosstalkSources_1125936_5.stderr parsl.dataflow.dflow INFO: Task 3176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1640c337e5f191f0d228a90bd55d8169 with result from task 3176 parsl.dataflow.dflow DEBUG: Task 3176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e0dd845f-2d89-4f15-8a4b-16cf29cba232_isrForCrosstalkSources_1125916_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e0dd845f-2d89-4f15-8a4b-16cf29cba232_isrForCrosstalkSources_1125916_50.stderr parsl.dataflow.dflow INFO: Task 3087 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 067d3a429920c2f6f2677387bc054621 with result from task 3087 parsl.dataflow.dflow DEBUG: Task 3087 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/0866168f-f4b4-4a98-a93c-7f318145c598_isrForCrosstalkSources_1125936_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/0866168f-f4b4-4a98-a93c-7f318145c598_isrForCrosstalkSources_1125936_37.stderr parsl.dataflow.dflow INFO: Task 3135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22c66a2418a3f537619c23e9dd3a65b3 with result from task 3135 parsl.dataflow.dflow DEBUG: Task 3135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/2ab51468-9bdf-4f9e-9000-1f6d2bbeddc8_isrForCrosstalkSources_1125892_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/2ab51468-9bdf-4f9e-9000-1f6d2bbeddc8_isrForCrosstalkSources_1125892_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:25 proc_lsst.multi:146 [INFO] found job 16896520 in provider checkpoint proc_lsst.multi INFO: found job 16896520 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f1375a286123304507e21329b896a79 with result from task 3162 parsl.dataflow.dflow DEBUG: Task 3162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/db6d9fd3-4546-4bc4-a7e7-80a3f56ddca2_isrForCrosstalkSources_1125889_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/db6d9fd3-4546-4bc4-a7e7-80a3f56ddca2_isrForCrosstalkSources_1125889_30.stderr parsl.dataflow.dflow INFO: Task 3181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2685348201d672f0cb1470dcd27ad9a9 with result from task 3181 parsl.dataflow.dflow DEBUG: Task 3181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/9f59011d-8e10-4722-aa5a-f3dbeee381eb_isrForCrosstalkSources_1125912_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/9f59011d-8e10-4722-aa5a-f3dbeee381eb_isrForCrosstalkSources_1125912_22.stderr parsl.dataflow.dflow INFO: Task 3195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1be206e5171e9fbe9cd7dee9c071db1 with result from task 3195 parsl.dataflow.dflow DEBUG: Task 3195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/fe65b84d-bb08-420f-8580-3eaeb59e4a81_isrForCrosstalkSources_1125939_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/fe65b84d-bb08-420f-8580-3eaeb59e4a81_isrForCrosstalkSources_1125939_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:25 proc_lsst.multi:146 [INFO] found job 16896521 in provider checkpoint proc_lsst.multi INFO: found job 16896521 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ba808df2fb26a35ca3224288cca203d with result from task 3167 parsl.dataflow.dflow DEBUG: Task 3167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/c349fd1d-1938-4049-9d52-d1c36a57788b_isrForCrosstalkSources_1125908_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/c349fd1d-1938-4049-9d52-d1c36a57788b_isrForCrosstalkSources_1125908_29.stderr parsl.dataflow.dflow INFO: Task 2865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e181a7daa2d99a2de5425647ad7a1ca with result from task 2865 parsl.dataflow.dflow DEBUG: Task 2865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/4d1b0e6f-f99b-4f29-8215-5c4fc99dff16_isrForCrosstalkSources_1125899_42.stdout parsl.dataflow.dflow INFO: Standard error for task 2865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/4d1b0e6f-f99b-4f29-8215-5c4fc99dff16_isrForCrosstalkSources_1125899_42.stderr parsl.dataflow.dflow INFO: Task 3002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5ed477afdc499031da1676e83de45c5 with result from task 3002 parsl.dataflow.dflow DEBUG: Task 3002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/3c976cc0-e270-4ce2-8a3d-eef44f61f278_isrForCrosstalkSources_1125903_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/3c976cc0-e270-4ce2-8a3d-eef44f61f278_isrForCrosstalkSources_1125903_58.stderr parsl.dataflow.dflow INFO: Task 3228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4e5a78b798898e43fc2989babdb6814 with result from task 3228 parsl.dataflow.dflow DEBUG: Task 3228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/5ea115ff-61df-4e77-b650-a343924d47b3_isrForCrosstalkSources_1125913_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/5ea115ff-61df-4e77-b650-a343924d47b3_isrForCrosstalkSources_1125913_49.stderr parsl.dataflow.dflow INFO: Task 3163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffe7426da8dc5fc6faddac356f18d09c with result from task 3163 parsl.dataflow.dflow DEBUG: Task 3163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/706d7fcb-d21b-4b03-a387-a7a49d0702ca_isrForCrosstalkSources_1125942_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/706d7fcb-d21b-4b03-a387-a7a49d0702ca_isrForCrosstalkSources_1125942_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:25 proc_lsst.multi:146 [INFO] found job 16896522 in provider checkpoint proc_lsst.multi INFO: found job 16896522 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80ff6edf8e3a4cc932ab95da620ff004 with result from task 3174 parsl.dataflow.dflow DEBUG: Task 3174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/6530a0cf-131f-488a-b254-6e1917c2f6a7_isrForCrosstalkSources_1125901_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/6530a0cf-131f-488a-b254-6e1917c2f6a7_isrForCrosstalkSources_1125901_17.stderr parsl.dataflow.dflow INFO: Task 3142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f33e2d7defe66815cd62c70da5cb528 with result from task 3142 parsl.dataflow.dflow DEBUG: Task 3142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/0d2a6833-9b37-408a-b192-09ca7a9f9680_isrForCrosstalkSources_1125929_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/0d2a6833-9b37-408a-b192-09ca7a9f9680_isrForCrosstalkSources_1125929_34.stderr parsl.dataflow.dflow INFO: Task 3175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d4ff28420521cc5a49ae582ef7c90eb with result from task 3175 parsl.dataflow.dflow DEBUG: Task 3175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/23bba5a2-fbcd-41c6-96f7-3c526e448cd9_isrForCrosstalkSources_1125906_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/23bba5a2-fbcd-41c6-96f7-3c526e448cd9_isrForCrosstalkSources_1125906_9.stderr parsl.dataflow.dflow INFO: Task 3190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d48a4328b58b81b4d492bbc42d08ca75 with result from task 3190 parsl.dataflow.dflow DEBUG: Task 3190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d440bd6f-5cfb-4a02-b2a4-7ccf89646aeb_isrForCrosstalkSources_1125911_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d440bd6f-5cfb-4a02-b2a4-7ccf89646aeb_isrForCrosstalkSources_1125911_62.stderr parsl.dataflow.dflow INFO: Task 3194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a04a4eb3278449505b518d0ec2862e6 with result from task 3194 parsl.dataflow.dflow DEBUG: Task 3194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/ac61d890-8f07-4ec2-8c97-f43803ea04fb_isrForCrosstalkSources_1125933_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/ac61d890-8f07-4ec2-8c97-f43803ea04fb_isrForCrosstalkSources_1125933_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:26 proc_lsst.multi:146 [INFO] found job 16896523 in provider checkpoint proc_lsst.multi INFO: found job 16896523 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b654307219d51119ceebf38409d6513 with result from task 3191 parsl.dataflow.dflow DEBUG: Task 3191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/7ba76eca-3d33-4155-8c35-9d1276dd106b_isrForCrosstalkSources_1125922_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/7ba76eca-3d33-4155-8c35-9d1276dd106b_isrForCrosstalkSources_1125922_31.stderr parsl.dataflow.dflow INFO: Task 3207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac9e917023d7a2a71fe00c317d1fe121 with result from task 3207 parsl.dataflow.dflow DEBUG: Task 3207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/92fef1c3-3012-4035-9868-8b9b35aa9986_isrForCrosstalkSources_1125925_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/92fef1c3-3012-4035-9868-8b9b35aa9986_isrForCrosstalkSources_1125925_49.stderr parsl.dataflow.dflow INFO: Task 3182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee5b320007bb6e0156bc46f9424cba34 with result from task 3182 parsl.dataflow.dflow DEBUG: Task 3182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/5b92f826-772d-4c39-81bf-27612ec51945_isrForCrosstalkSources_1125943_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/5b92f826-772d-4c39-81bf-27612ec51945_isrForCrosstalkSources_1125943_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:26 proc_lsst.multi:146 [INFO] found job 16896524 in provider checkpoint proc_lsst.multi INFO: found job 16896524 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e511434977dc335d4bd07c65abd31d3 with result from task 3198 parsl.dataflow.dflow DEBUG: Task 3198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/31f4b82e-6fe1-4d67-a8a5-ba4f70bc2ffe_isrForCrosstalkSources_1125914_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/31f4b82e-6fe1-4d67-a8a5-ba4f70bc2ffe_isrForCrosstalkSources_1125914_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:26 proc_lsst.multi:146 [INFO] found job 16896525 in provider checkpoint proc_lsst.multi INFO: found job 16896525 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f93e44d90023425171618358aba33341 with result from task 3160 parsl.dataflow.dflow DEBUG: Task 3160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/31fc3117-7247-40ed-8e4e-7075f614b382_isrForCrosstalkSources_1125919_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/31fc3117-7247-40ed-8e4e-7075f614b382_isrForCrosstalkSources_1125919_13.stderr parsl.dataflow.dflow INFO: Task 3264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 965c98c9fe206771dbab0619599e496e with result from task 3264 parsl.dataflow.dflow DEBUG: Task 3264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/fb75d4ab-2478-4443-90d7-67a983917978_isrForCrosstalkSources_1125912_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/fb75d4ab-2478-4443-90d7-67a983917978_isrForCrosstalkSources_1125912_21.stderr parsl.dataflow.dflow INFO: Task 3124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bcf611d2bee15f9dad62ac7adee7675 with result from task 3124 parsl.dataflow.dflow DEBUG: Task 3124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/73a97a49-ade6-4bbd-abb3-89415db73ca3_isrForCrosstalkSources_1125939_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/73a97a49-ade6-4bbd-abb3-89415db73ca3_isrForCrosstalkSources_1125939_33.stderr parsl.dataflow.dflow INFO: Task 3222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 977f8e31af44f5ea89348e283961c7ce with result from task 3222 parsl.dataflow.dflow DEBUG: Task 3222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/9dda99eb-3446-4185-91a0-7bf582966f2b_isrForCrosstalkSources_1125930_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/9dda99eb-3446-4185-91a0-7bf582966f2b_isrForCrosstalkSources_1125930_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:27 proc_lsst.multi:146 [INFO] found job 16896526 in provider checkpoint proc_lsst.multi INFO: found job 16896526 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42217e1b45afa695754ac6c3cea1f05a with result from task 3216 parsl.dataflow.dflow DEBUG: Task 3216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/4e88c3c2-1d5a-4f6e-baf0-956445eddfde_isrForCrosstalkSources_1125903_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/4e88c3c2-1d5a-4f6e-baf0-956445eddfde_isrForCrosstalkSources_1125903_13.stderr parsl.dataflow.dflow INFO: Task 3279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c0d989f3b0e8eb2e53a78ded61d60d5 with result from task 3279 parsl.dataflow.dflow DEBUG: Task 3279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/91c73496-b541-46e0-82b6-5d3e998764d3_isrForCrosstalkSources_1125896_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/91c73496-b541-46e0-82b6-5d3e998764d3_isrForCrosstalkSources_1125896_51.stderr parsl.dataflow.dflow INFO: Task 3265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d81e1e9bc56384cd551903ddd54bc8e with result from task 3265 parsl.dataflow.dflow DEBUG: Task 3265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/70279cff-afb4-4cc4-a07f-1c0bd69fd594_isrForCrosstalkSources_1125913_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/70279cff-afb4-4cc4-a07f-1c0bd69fd594_isrForCrosstalkSources_1125913_33.stderr parsl.dataflow.dflow INFO: Task 3287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d6dfc370b5c34a10a47b0e070a6c7c3 with result from task 3287 parsl.dataflow.dflow DEBUG: Task 3287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/9be8919e-f10b-458d-8f21-0702d78c6f80_isrForCrosstalkSources_1125905_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/9be8919e-f10b-458d-8f21-0702d78c6f80_isrForCrosstalkSources_1125905_14.stderr parsl.dataflow.dflow INFO: Task 3215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88bcea58280efb7f744c89a5a3a1d7cc with result from task 3215 parsl.dataflow.dflow DEBUG: Task 3215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/00214485-930f-4b12-a44a-d60cc48ae2ce_isrForCrosstalkSources_1125930_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/00214485-930f-4b12-a44a-d60cc48ae2ce_isrForCrosstalkSources_1125930_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:27 proc_lsst.multi:146 [INFO] found job 16896527 in provider checkpoint proc_lsst.multi INFO: found job 16896527 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38f35800b9c13dadfb29bb861f2b74dc with result from task 3263 parsl.dataflow.dflow DEBUG: Task 3263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/f54a8f16-3654-4af7-9ec5-61f25d27a3d1_isrForCrosstalkSources_1125937_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/f54a8f16-3654-4af7-9ec5-61f25d27a3d1_isrForCrosstalkSources_1125937_11.stderr parsl.dataflow.dflow INFO: Task 3192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6f6528a6cfe75d11c4e6658ed17c62a with result from task 3192 parsl.dataflow.dflow DEBUG: Task 3192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/612a85d0-c8e5-4143-8a89-912fac6e1105_isrForCrosstalkSources_1125926_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/612a85d0-c8e5-4143-8a89-912fac6e1105_isrForCrosstalkSources_1125926_2.stderr parsl.dataflow.dflow INFO: Task 3240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1820e5cc10d819191e5207d2c06339f with result from task 3240 parsl.dataflow.dflow DEBUG: Task 3240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/b21cc217-a015-4e99-91ed-01c67eb6c6af_isrForCrosstalkSources_1125890_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/b21cc217-a015-4e99-91ed-01c67eb6c6af_isrForCrosstalkSources_1125890_22.stderr parsl.dataflow.dflow INFO: Task 3210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4657e1b6c6e1961de3fb8f39c7a3ccc6 with result from task 3210 parsl.dataflow.dflow DEBUG: Task 3210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/4c964d66-0d75-464a-ba94-9d375aa0e8f5_isrForCrosstalkSources_1125926_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/4c964d66-0d75-464a-ba94-9d375aa0e8f5_isrForCrosstalkSources_1125926_34.stderr parsl.dataflow.dflow INFO: Task 3246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01ae455b0875930fd048e91326446068 with result from task 3246 parsl.dataflow.dflow DEBUG: Task 3246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e3063182-1282-4a5f-acc0-eb77abef087b_isrForCrosstalkSources_1125895_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/e3063182-1282-4a5f-acc0-eb77abef087b_isrForCrosstalkSources_1125895_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3242 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 8443f383a7b241e882e4087f74c3b819 with result from task 3242 parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3242 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/cca6fdd2-0956-45ad-bdaf-910e66384e0b_isrForCrosstalkSources_1125921_4.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/cca6fdd2-0956-45ad-bdaf-910e66384e0b_isrForCrosstalkSources_1125921_4.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:27 proc_lsst.multi:146 [INFO] found job 16896528 in provider checkpoint proc_lsst.multi INFO: found job 16896528 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 837d71b83f50c9e9eaf80137dceecdfc with result from task 3196 parsl.dataflow.dflow DEBUG: Task 3196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/8a4a6994-aa58-44ae-a7a9-6995def2f5a6_isrForCrosstalkSources_1125890_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/8a4a6994-aa58-44ae-a7a9-6995def2f5a6_isrForCrosstalkSources_1125890_28.stderr parsl.dataflow.dflow INFO: Task 3278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32d2ccd70abe651b19a521e9e81b8110 with result from task 3278 parsl.dataflow.dflow DEBUG: Task 3278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/96a2065b-6818-45a6-a096-2f2fb29ece9f_isrForCrosstalkSources_1125940_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/96a2065b-6818-45a6-a096-2f2fb29ece9f_isrForCrosstalkSources_1125940_13.stderr parsl.dataflow.dflow INFO: Task 3178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7063d2feb01a8ff792a5e58b3fa8d23f with result from task 3178 parsl.dataflow.dflow DEBUG: Task 3178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/33504d93-8ac3-4faf-ac12-f716fdcfdd95_isrForCrosstalkSources_1125905_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/33504d93-8ac3-4faf-ac12-f716fdcfdd95_isrForCrosstalkSources_1125905_46.stderr parsl.dataflow.dflow INFO: Task 3295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80aa53196123728a7710a6e1dd172e91 with result from task 3295 parsl.dataflow.dflow DEBUG: Task 3295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/58e112fe-7370-4e33-82f0-a9681bc78a72_isrForCrosstalkSources_1125939_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/58e112fe-7370-4e33-82f0-a9681bc78a72_isrForCrosstalkSources_1125939_22.stderr parsl.dataflow.dflow INFO: Task 2878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de14f9490932be94abb5cc9d69e0fc6d with result from task 2878 parsl.dataflow.dflow DEBUG: Task 2878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/3b4dfc17-edd3-4611-b127-d85ed31f07f3_isrForCrosstalkSources_1125914_29.stdout parsl.dataflow.dflow INFO: Standard error for task 2878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/3b4dfc17-edd3-4611-b127-d85ed31f07f3_isrForCrosstalkSources_1125914_29.stderr parsl.dataflow.dflow INFO: Task 3177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3b1cf2fe8d138dc5edb4eee1c184726 with result from task 3177 parsl.dataflow.dflow DEBUG: Task 3177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/482f6b11-1163-4135-ae88-3eb4f2f6115f_isrForCrosstalkSources_1125903_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/482f6b11-1163-4135-ae88-3eb4f2f6115f_isrForCrosstalkSources_1125903_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:27 proc_lsst.multi:146 [INFO] found job 16896529 in provider checkpoint proc_lsst.multi INFO: found job 16896529 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a52a7b47df78d219a28f8a2bd20a59c0 with result from task 3164 parsl.dataflow.dflow DEBUG: Task 3164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/e5ccbdd7-5272-4575-9771-e20fec97cf1f_isrForCrosstalkSources_1125938_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/e5ccbdd7-5272-4575-9771-e20fec97cf1f_isrForCrosstalkSources_1125938_14.stderr parsl.dataflow.dflow INFO: Task 3218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4e1b8f6f6b5df69732ea8053b44e120 with result from task 3218 parsl.dataflow.dflow DEBUG: Task 3218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/452bccb1-717b-46b2-8698-bde8847ae5cf_isrForCrosstalkSources_1125930_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/452bccb1-717b-46b2-8698-bde8847ae5cf_isrForCrosstalkSources_1125930_36.stderr parsl.dataflow.dflow INFO: Task 3238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bab9aa23b4e0a392fa4f57009a74528 with result from task 3238 parsl.dataflow.dflow DEBUG: Task 3238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/1aabc82b-1c0b-4d3c-b2ab-851c26f271b7_isrForCrosstalkSources_1125913_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/1aabc82b-1c0b-4d3c-b2ab-851c26f271b7_isrForCrosstalkSources_1125913_34.stderr parsl.dataflow.dflow INFO: Task 3259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c79d95e92fc1d244c61dda41a84f086c with result from task 3259 parsl.dataflow.dflow DEBUG: Task 3259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c1f31d53-9273-43cb-82d7-42083aa61d3d_isrForCrosstalkSources_1125893_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/c1f31d53-9273-43cb-82d7-42083aa61d3d_isrForCrosstalkSources_1125893_17.stderr parsl.dataflow.dflow INFO: Task 3285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8347e15c0361f80c13415accf20d02c8 with result from task 3285 parsl.dataflow.dflow DEBUG: Task 3285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/614989e7-d633-4686-99ca-299330c827c0_isrForCrosstalkSources_1125905_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/614989e7-d633-4686-99ca-299330c827c0_isrForCrosstalkSources_1125905_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:27 proc_lsst.multi:146 [INFO] found job 16896530 in provider checkpoint proc_lsst.multi INFO: found job 16896530 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f0ca6cedf066353037b431a138a61a4 with result from task 3213 parsl.dataflow.dflow DEBUG: Task 3213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/b59242c7-f11d-4b71-bbd3-a540fb055fa6_isrForCrosstalkSources_1125905_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/b59242c7-f11d-4b71-bbd3-a540fb055fa6_isrForCrosstalkSources_1125905_39.stderr parsl.dataflow.dflow INFO: Task 3254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53b6e9d112b4b18cc80af07c26509809 with result from task 3254 parsl.dataflow.dflow DEBUG: Task 3254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e141419e-9aa5-4f31-aeab-d2614d09a1ff_isrForCrosstalkSources_1125937_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e141419e-9aa5-4f31-aeab-d2614d09a1ff_isrForCrosstalkSources_1125937_28.stderr parsl.dataflow.dflow INFO: Task 3289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5693db3db404822b7d615dbf2e64678a with result from task 3289 parsl.dataflow.dflow DEBUG: Task 3289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/dfd2e7fc-1c9f-42a3-8713-b993a633e6c0_isrForCrosstalkSources_1125910_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/dfd2e7fc-1c9f-42a3-8713-b993a633e6c0_isrForCrosstalkSources_1125910_45.stderr parsl.dataflow.dflow INFO: Task 3267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7b7527902dd56e67ee0cb4192a16004 with result from task 3267 parsl.dataflow.dflow DEBUG: Task 3267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/8b871836-658b-4bfc-807e-f9d10e685bd7_isrForCrosstalkSources_1125928_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/8b871836-658b-4bfc-807e-f9d10e685bd7_isrForCrosstalkSources_1125928_23.stderr parsl.dataflow.dflow INFO: Task 3266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4727408ef66b213b01634b6273eefd2e with result from task 3266 parsl.dataflow.dflow DEBUG: Task 3266 checkpointed parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:28 proc_lsst.multi:146 [INFO] found job 16896531 in provider checkpoint proc_lsst.multi INFO: found job 16896531 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/9c092ac2-6d24-482a-844b-124c96c93a61_isrForCrosstalkSources_1125937_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/9c092ac2-6d24-482a-844b-124c96c93a61_isrForCrosstalkSources_1125937_1.stderr parsl.dataflow.dflow INFO: Task 3206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e861ff95d17a588981a31f0b9648c87a with result from task 3206 parsl.dataflow.dflow DEBUG: Task 3206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/914e2c46-130c-49f4-acf5-3ea2fe0b4b8c_isrForCrosstalkSources_1125937_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/914e2c46-130c-49f4-acf5-3ea2fe0b4b8c_isrForCrosstalkSources_1125937_45.stderr parsl.dataflow.dflow INFO: Task 3202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38a9956c7fc526552a7d01369df7958e with result from task 3202 parsl.dataflow.dflow DEBUG: Task 3202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/67022975-7b1f-46a7-9a71-f020e7fb25f9_isrForCrosstalkSources_1125892_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/67022975-7b1f-46a7-9a71-f020e7fb25f9_isrForCrosstalkSources_1125892_48.stderr parsl.dataflow.dflow INFO: Task 3300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dd3e4649ebfb088842001fbbbc74247 with result from task 3300 parsl.dataflow.dflow DEBUG: Task 3300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/7f540807-fc30-4e49-901c-b034977be9c6_isrForCrosstalkSources_1125931_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/7f540807-fc30-4e49-901c-b034977be9c6_isrForCrosstalkSources_1125931_26.stderr parsl.dataflow.dflow INFO: Task 3226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc314f4cee646baf1ab6cc10294df1fe with result from task 3226 parsl.dataflow.dflow DEBUG: Task 3226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4121790e-2e67-44f6-bd17-c1f34e0713c1_isrForCrosstalkSources_1125893_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/4121790e-2e67-44f6-bd17-c1f34e0713c1_isrForCrosstalkSources_1125893_23.stderr parsl.dataflow.dflow INFO: Task 3260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f61345224af95fb104c96d5b0959ffe4 with result from task 3260 parsl.dataflow.dflow DEBUG: Task 3260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/0e86816c-37cd-4284-afe0-c6b674d3c7a1_isrForCrosstalkSources_1125888_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/0e86816c-37cd-4284-afe0-c6b674d3c7a1_isrForCrosstalkSources_1125888_13.stderr parsl.dataflow.dflow INFO: Task 3252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a46553780df80d9a7ddcc1dd9f431b18 with result from task 3252 parsl.dataflow.dflow DEBUG: Task 3252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/a6572254-5dda-4af7-b4a1-d81e8b3163d9_isrForCrosstalkSources_1125946_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/a6572254-5dda-4af7-b4a1-d81e8b3163d9_isrForCrosstalkSources_1125946_13.stderr parsl.dataflow.dflow INFO: Task 3235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3efbafcff59dd0be0cbb8bb028b135a0 with result from task 3235 parsl.dataflow.dflow DEBUG: Task 3235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/6e99b67f-1ec7-40c6-8ed9-cadc73e3154b_isrForCrosstalkSources_1125944_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/6e99b67f-1ec7-40c6-8ed9-cadc73e3154b_isrForCrosstalkSources_1125944_51.stderr parsl.dataflow.dflow INFO: Task 3200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3605ac668728013941d871270d25fbb1 with result from task 3200 parsl.dataflow.dflow DEBUG: Task 3200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/09a0aee3-3cfc-4e3b-ae30-628a4872896b_isrForCrosstalkSources_1125931_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/09a0aee3-3cfc-4e3b-ae30-628a4872896b_isrForCrosstalkSources_1125931_15.stderr parsl.dataflow.dflow INFO: Task 3303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28032acc2af8fd2b1ef0dd95f88c4af0 with result from task 3303 parsl.dataflow.dflow DEBUG: Task 3303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/a8e601ad-5235-4acf-aafa-ba9e7e4e858e_isrForCrosstalkSources_1125889_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/a8e601ad-5235-4acf-aafa-ba9e7e4e858e_isrForCrosstalkSources_1125889_23.stderr parsl.dataflow.dflow INFO: Task 3221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbed9c94612e7c00cdb95c66ba261249 with result from task 3221 parsl.dataflow.dflow DEBUG: Task 3221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/85ee3770-afd2-4252-a08d-b1f458d028c6_isrForCrosstalkSources_1125934_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/85ee3770-afd2-4252-a08d-b1f458d028c6_isrForCrosstalkSources_1125934_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:28 proc_lsst.multi:146 [INFO] found job 16896533 in provider checkpoint proc_lsst.multi INFO: found job 16896533 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d8c2b9c0e16a7fef06f3c9b361fb1cf with result from task 3219 parsl.dataflow.dflow DEBUG: Task 3219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/e4a00982-56b2-4bff-ac60-ed615dfb7b40_isrForCrosstalkSources_1125890_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/e4a00982-56b2-4bff-ac60-ed615dfb7b40_isrForCrosstalkSources_1125890_15.stderr parsl.dataflow.dflow INFO: Task 3232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 540ca4207b832d5d1e885f29cee6e079 with result from task 3232 parsl.dataflow.dflow DEBUG: Task 3232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/1ee99885-f432-4941-a3ce-1903154d7178_isrForCrosstalkSources_1125911_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/1ee99885-f432-4941-a3ce-1903154d7178_isrForCrosstalkSources_1125911_59.stderr parsl.dataflow.dflow INFO: Task 3270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0ceca61ff0feea1b97d8d55557bfc5a with result from task 3270 parsl.dataflow.dflow DEBUG: Task 3270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4aa09ddf-7153-4dec-a861-958eb78e6c69_isrForCrosstalkSources_1125908_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/4aa09ddf-7153-4dec-a861-958eb78e6c69_isrForCrosstalkSources_1125908_8.stderr parsl.dataflow.dflow INFO: Task 3243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbaa50a762ed2c4bb018dfb9726ee41f with result from task 3243 parsl.dataflow.dflow DEBUG: Task 3243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/58811115-28f0-42f5-8dab-5cd3c8f983ec_isrForCrosstalkSources_1125915_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/58811115-28f0-42f5-8dab-5cd3c8f983ec_isrForCrosstalkSources_1125915_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:28 proc_lsst.multi:146 [INFO] found job 16896534 in provider checkpoint proc_lsst.multi INFO: found job 16896534 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6aa2528dd8559e1cc3fa092a5d346c51 with result from task 3276 parsl.dataflow.dflow DEBUG: Task 3276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3a3606e8-1bef-4491-b19c-3fd021419755_isrForCrosstalkSources_1125920_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3a3606e8-1bef-4491-b19c-3fd021419755_isrForCrosstalkSources_1125920_47.stderr parsl.dataflow.dflow INFO: Task 3311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a66d28d270bd285d3b21611df8c943d with result from task 3311 parsl.dataflow.dflow DEBUG: Task 3311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/51cef015-83d2-4237-9b8b-e749d94b8a71_isrForCrosstalkSources_1125919_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/51cef015-83d2-4237-9b8b-e749d94b8a71_isrForCrosstalkSources_1125919_51.stderr parsl.dataflow.dflow INFO: Task 3308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d31d14ed7205724c48d72d240d003c64 with result from task 3308 parsl.dataflow.dflow DEBUG: Task 3308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/82265d5f-d1b1-4004-b997-0fd41f59fbf6_isrForCrosstalkSources_1125888_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/82265d5f-d1b1-4004-b997-0fd41f59fbf6_isrForCrosstalkSources_1125888_34.stderr parsl.dataflow.dflow INFO: Task 3313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 157a1d1c2055651094ee3f9396d30c59 with result from task 3313 parsl.dataflow.dflow DEBUG: Task 3313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d2d81f77-1097-44eb-ae57-6499601d74b3_isrForCrosstalkSources_1125919_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d2d81f77-1097-44eb-ae57-6499601d74b3_isrForCrosstalkSources_1125919_19.stderr parsl.dataflow.dflow INFO: Task 3282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 624772d35e35d9422ec23845da5b3530 with result from task 3282 parsl.dataflow.dflow DEBUG: Task 3282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/fbd3f7b4-2ec0-4735-8d94-db8258f2e21f_isrForCrosstalkSources_1125937_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/fbd3f7b4-2ec0-4735-8d94-db8258f2e21f_isrForCrosstalkSources_1125937_33.stderr parsl.dataflow.dflow INFO: Task 3268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4546548114b13a650d1f64cc30b25464 with result from task 3268 parsl.dataflow.dflow DEBUG: Task 3268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/8dc859af-5e59-427c-adc2-6b0d8f4207d5_isrForCrosstalkSources_1125904_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/8dc859af-5e59-427c-adc2-6b0d8f4207d5_isrForCrosstalkSources_1125904_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:28 proc_lsst.multi:146 [INFO] found job 16896535 in provider checkpoint proc_lsst.multi INFO: found job 16896535 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff9d36bbeef647306f29673ea0b9164c with result from task 3241 parsl.dataflow.dflow DEBUG: Task 3241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/88168534-f2fd-485e-b28e-68ee9811f893_isrForCrosstalkSources_1125914_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/88168534-f2fd-485e-b28e-68ee9811f893_isrForCrosstalkSources_1125914_24.stderr parsl.dataflow.dflow INFO: Task 3340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ffc30607f98506d193c4bb401ee0ee1 with result from task 3340 parsl.dataflow.dflow DEBUG: Task 3340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/38fd74cc-db59-4505-86e7-863b337e1451_isrForCrosstalkSources_1125937_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/38fd74cc-db59-4505-86e7-863b337e1451_isrForCrosstalkSources_1125937_27.stderr parsl.dataflow.dflow INFO: Task 3316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cd3f172455b33bdaa7aaadace25b349 with result from task 3316 parsl.dataflow.dflow DEBUG: Task 3316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/48a46a6d-41d9-4f02-bd56-f2005a488423_isrForCrosstalkSources_1125889_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/48a46a6d-41d9-4f02-bd56-f2005a488423_isrForCrosstalkSources_1125889_62.stderr parsl.dataflow.dflow INFO: Task 3325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79cda1b6b69e43f0b24dec637b9ef47d with result from task 3325 parsl.dataflow.dflow DEBUG: Task 3325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/8e5c5d40-a348-4923-b0e7-aa7dced57321_isrForCrosstalkSources_1125918_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/8e5c5d40-a348-4923-b0e7-aa7dced57321_isrForCrosstalkSources_1125918_26.stderr parsl.dataflow.dflow INFO: Task 3257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 035a6278b07fc422fb5a43f478479345 with result from task 3257 parsl.dataflow.dflow DEBUG: Task 3257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/13790a49-9050-4c2c-93e8-9640f21cda5c_isrForCrosstalkSources_1125908_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/13790a49-9050-4c2c-93e8-9640f21cda5c_isrForCrosstalkSources_1125908_11.stderr parsl.dataflow.dflow INFO: Task 3249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de89b77c081f9ed6caa451646662f0bc with result from task 3249 parsl.dataflow.dflow DEBUG: Task 3249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b06ef118-70ff-478f-a82f-94c60063df10_isrForCrosstalkSources_1125938_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/b06ef118-70ff-478f-a82f-94c60063df10_isrForCrosstalkSources_1125938_48.stderr parsl.dataflow.dflow INFO: Task 3346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00cc3bfe88257fc6b38cfa2834af3465 with result from task 3346 parsl.dataflow.dflow DEBUG: Task 3346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/5595126d-26e1-44a6-90a0-eabee85ff750_isrForCrosstalkSources_1125928_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/5595126d-26e1-44a6-90a0-eabee85ff750_isrForCrosstalkSources_1125928_56.stderr parsl.dataflow.dflow INFO: Task 3250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93d1775aedf58d739a21ae35518942c9 with result from task 3250 parsl.dataflow.dflow DEBUG: Task 3250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/f286d2cf-afa6-4704-aa8b-6c76e4edc359_isrForCrosstalkSources_1125941_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/f286d2cf-afa6-4704-aa8b-6c76e4edc359_isrForCrosstalkSources_1125941_58.stderr parsl.dataflow.dflow INFO: Task 3342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b492faba184ba16e59df8ac064fd502 with result from task 3342 parsl.dataflow.dflow DEBUG: Task 3342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/29e462be-b420-4386-b714-e23ab146404e_isrForCrosstalkSources_1125900_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/29e462be-b420-4386-b714-e23ab146404e_isrForCrosstalkSources_1125900_27.stderr parsl.dataflow.dflow INFO: Task 3321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cef15e9bba015a69c20553439ee01e6 with result from task 3321 parsl.dataflow.dflow DEBUG: Task 3321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/2120f660-6624-4cbf-a0c1-bf9bf2577650_isrForCrosstalkSources_1125930_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/2120f660-6624-4cbf-a0c1-bf9bf2577650_isrForCrosstalkSources_1125930_24.stderr parsl.dataflow.dflow INFO: Task 3315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33fd8cbb26005ac0aa5c238fd49f534d with result from task 3315 parsl.dataflow.dflow DEBUG: Task 3315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/ace305ff-3534-4aa5-b237-f547560b66f0_isrForCrosstalkSources_1125927_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/ace305ff-3534-4aa5-b237-f547560b66f0_isrForCrosstalkSources_1125927_54.stderr parsl.dataflow.dflow INFO: Task 3185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28812615dec67fc78e993674abda6aa2 with result from task 3185 parsl.dataflow.dflow DEBUG: Task 3185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/9461ed30-7de8-4cab-9a18-4ae5f90a75ac_isrForCrosstalkSources_1125919_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/9461ed30-7de8-4cab-9a18-4ae5f90a75ac_isrForCrosstalkSources_1125919_23.stderr parsl.dataflow.dflow INFO: Task 3223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f89db9a92333b0641ad520f6a0c77dde with result from task 3223 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3223 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:29 proc_lsst.multi:146 [INFO] found job 16896536 in provider checkpoint proc_lsst.multi INFO: found job 16896536 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a3a0ac6b-7f18-47d1-bed5-d0b97cd95ccd_isrForCrosstalkSources_1125921_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a3a0ac6b-7f18-47d1-bed5-d0b97cd95ccd_isrForCrosstalkSources_1125921_59.stderr parsl.dataflow.dflow INFO: Task 3189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5754f7dcc71a7ec19f9b597fd69d51af with result from task 3189 parsl.dataflow.dflow DEBUG: Task 3189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/103bae7b-67d9-4b4b-b39c-c924ceb9c0d1_isrForCrosstalkSources_1125938_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/103bae7b-67d9-4b4b-b39c-c924ceb9c0d1_isrForCrosstalkSources_1125938_9.stderr parsl.dataflow.dflow INFO: Task 3225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fcac2580eac7f0d4a76e103ecadb4d7 with result from task 3225 parsl.dataflow.dflow DEBUG: Task 3225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c89f5f34-3d24-4d1e-af47-f956ef18b25e_isrForCrosstalkSources_1125904_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/c89f5f34-3d24-4d1e-af47-f956ef18b25e_isrForCrosstalkSources_1125904_54.stderr parsl.dataflow.dflow INFO: Task 3256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23d11dd7204c009efe367f8b85c7559f with result from task 3256 parsl.dataflow.dflow DEBUG: Task 3256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/ba5686f3-a0c3-4e9f-9935-82e630ce069a_isrForCrosstalkSources_1125893_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/ba5686f3-a0c3-4e9f-9935-82e630ce069a_isrForCrosstalkSources_1125893_42.stderr parsl.dataflow.dflow INFO: Task 3326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3827cd42fde5fd1740d6d738c0a84ef5 with result from task 3326 parsl.dataflow.dflow DEBUG: Task 3326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/131461e2-269c-432b-9a30-de59af3fa303_isrForCrosstalkSources_1125895_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/131461e2-269c-432b-9a30-de59af3fa303_isrForCrosstalkSources_1125895_13.stderr parsl.dataflow.dflow INFO: Task 3239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8158ce449668d93bf6b980e7e5d7b4b with result from task 3239 parsl.dataflow.dflow DEBUG: Task 3239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/3263f196-9f39-46ba-a92f-dbf72970a5c3_isrForCrosstalkSources_1125917_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/3263f196-9f39-46ba-a92f-dbf72970a5c3_isrForCrosstalkSources_1125917_16.stderr parsl.dataflow.dflow INFO: Task 3359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2993549bc291e1ee180f22ab8477c7d6 with result from task 3359 parsl.dataflow.dflow DEBUG: Task 3359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/e63112ed-724d-4bc4-b316-1c373e177e68_isrForCrosstalkSources_1125918_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/e63112ed-724d-4bc4-b316-1c373e177e68_isrForCrosstalkSources_1125918_22.stderr parsl.dataflow.dflow INFO: Task 3203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6c28905fbf0bb3e65e8ea84dbe242e9 with result from task 3203 parsl.dataflow.dflow DEBUG: Task 3203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/1d12fe05-7473-4835-aa8c-1e6cd8c8999b_isrForCrosstalkSources_1125928_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/1d12fe05-7473-4835-aa8c-1e6cd8c8999b_isrForCrosstalkSources_1125928_1.stderr parsl.dataflow.dflow INFO: Task 3341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f03aaa21f55c6e393abbf7e3bbff420f with result from task 3341 parsl.dataflow.dflow DEBUG: Task 3341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e52cd6db-9854-49b6-b8d6-cfab6ded6e9e_isrForCrosstalkSources_1125901_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/e52cd6db-9854-49b6-b8d6-cfab6ded6e9e_isrForCrosstalkSources_1125901_51.stderr parsl.dataflow.dflow INFO: Task 3201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8e97aec7c606f73391d6162f598a1ea with result from task 3201 parsl.dataflow.dflow DEBUG: Task 3201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d18451c3-7914-4978-9154-f3104ff82f52_isrForCrosstalkSources_1125911_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/d18451c3-7914-4978-9154-f3104ff82f52_isrForCrosstalkSources_1125911_13.stderr parsl.dataflow.dflow INFO: Task 3271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d9482f79352bb7ee213c56d64db80b8 with result from task 3271 parsl.dataflow.dflow DEBUG: Task 3271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/a790d3cd-6aa3-4f51-8e73-ff8344a6212d_isrForCrosstalkSources_1125890_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/a790d3cd-6aa3-4f51-8e73-ff8344a6212d_isrForCrosstalkSources_1125890_44.stderr parsl.dataflow.dflow INFO: Task 3348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00489d3dc30c31d07f474b4293f90b5d with result from task 3348 parsl.dataflow.dflow DEBUG: Task 3348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/90068e94-18a8-4ed1-930c-c77f2d0b65ec_isrForCrosstalkSources_1125907_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/90068e94-18a8-4ed1-930c-c77f2d0b65ec_isrForCrosstalkSources_1125907_4.stderr parsl.dataflow.dflow INFO: Task 3224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1741fb841037426b301b69a286afce90 with result from task 3224 parsl.dataflow.dflow DEBUG: Task 3224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c67a3946-44d9-4fe5-a85c-b5947268dc26_isrForCrosstalkSources_1125910_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/c67a3946-44d9-4fe5-a85c-b5947268dc26_isrForCrosstalkSources_1125910_38.stderr parsl.dataflow.dflow INFO: Task 3229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d1cd894906d2493d4801e9ab2f3808b with result from task 3229 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:30 proc_lsst.multi:146 [INFO] found job 16896537 in provider checkpoint proc_lsst.multi INFO: found job 16896537 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 3229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/d63aa3c0-bc93-4dbf-b47d-293fe39ead56_isrForCrosstalkSources_1125910_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/d63aa3c0-bc93-4dbf-b47d-293fe39ead56_isrForCrosstalkSources_1125910_3.stderr parsl.dataflow.dflow INFO: Task 3244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6df630428a6e61fb30a6f0ae90104caf with result from task 3244 parsl.dataflow.dflow DEBUG: Task 3244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/9ee490be-4b5b-4a10-b112-6257477e6b01_isrForCrosstalkSources_1125907_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/9ee490be-4b5b-4a10-b112-6257477e6b01_isrForCrosstalkSources_1125907_35.stderr parsl.dataflow.dflow INFO: Task 3220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec93581c5ae18f42c8b904ca4dd929dc with result from task 3220 parsl.dataflow.dflow DEBUG: Task 3220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/ef433f7e-a37e-4b40-a404-8c395df9a137_isrForCrosstalkSources_1125895_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/ef433f7e-a37e-4b40-a404-8c395df9a137_isrForCrosstalkSources_1125895_27.stderr parsl.dataflow.dflow INFO: Task 3350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c46fe4fd97089e18c990e132fa9b3c41 with result from task 3350 parsl.dataflow.dflow DEBUG: Task 3350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/08c84218-3932-4a12-831d-38176efeec10_isrForCrosstalkSources_1125910_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/08c84218-3932-4a12-831d-38176efeec10_isrForCrosstalkSources_1125910_36.stderr parsl.dataflow.dflow INFO: Task 3339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 423f27c556c71d4034fb73e17eb6f93b with result from task 3339 parsl.dataflow.dflow DEBUG: Task 3339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/9b080fb4-2eda-4de1-98c0-f9ce150aad58_isrForCrosstalkSources_1125894_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/9b080fb4-2eda-4de1-98c0-f9ce150aad58_isrForCrosstalkSources_1125894_46.stderr parsl.dataflow.dflow INFO: Task 3231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc3ecb9b5069d9ca1e5f58c872b41d4a with result from task 3231 parsl.dataflow.dflow DEBUG: Task 3231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/7578d3bb-a0f3-4173-95ae-47bfc3c12fd2_isrForCrosstalkSources_1125889_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/7578d3bb-a0f3-4173-95ae-47bfc3c12fd2_isrForCrosstalkSources_1125889_38.stderr parsl.dataflow.dflow INFO: Task 3322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40a435902b2f786dda23c5b963235765 with result from task 3322 parsl.dataflow.dflow DEBUG: Task 3322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/aee262ce-68d8-43f3-a6be-460100dc7c8d_isrForCrosstalkSources_1125928_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/aee262ce-68d8-43f3-a6be-460100dc7c8d_isrForCrosstalkSources_1125928_6.stderr parsl.dataflow.dflow INFO: Task 3290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3fec94400acc14f0600ad23b566ed2c with result from task 3290 parsl.dataflow.dflow DEBUG: Task 3290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/bbfd1090-92aa-4f08-a037-07c1a18b43a6_isrForCrosstalkSources_1125910_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/bbfd1090-92aa-4f08-a037-07c1a18b43a6_isrForCrosstalkSources_1125910_14.stderr parsl.dataflow.dflow INFO: Task 3362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93041748238e31f3ff721c5092f9d138 with result from task 3362 parsl.dataflow.dflow DEBUG: Task 3362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/7290d2a9-f378-4b1a-ac5c-107a4f1583d9_isrForCrosstalkSources_1125890_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/7290d2a9-f378-4b1a-ac5c-107a4f1583d9_isrForCrosstalkSources_1125890_34.stderr parsl.dataflow.dflow INFO: Task 3251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13de7e92082c476880d29ad98d7ccf08 with result from task 3251 parsl.dataflow.dflow DEBUG: Task 3251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/a40e3d07-3229-4942-9497-375ad8663182_isrForCrosstalkSources_1125892_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/a40e3d07-3229-4942-9497-375ad8663182_isrForCrosstalkSources_1125892_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:30 proc_lsst.multi:146 [INFO] found job 16896538 in provider checkpoint proc_lsst.multi INFO: found job 16896538 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a545642e77ff1313125792b37d696d5b with result from task 3234 parsl.dataflow.dflow DEBUG: Task 3234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/808aa5bc-dbca-453a-a23f-c98065d4b1a4_isrForCrosstalkSources_1125938_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/808aa5bc-dbca-453a-a23f-c98065d4b1a4_isrForCrosstalkSources_1125938_1.stderr parsl.dataflow.dflow INFO: Task 3354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f6be80d1f726b1f3874fd82eb70fe18 with result from task 3354 parsl.dataflow.dflow DEBUG: Task 3354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/89584e9e-3b70-48f8-83cd-31aa81e834d0_isrForCrosstalkSources_1125923_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/89584e9e-3b70-48f8-83cd-31aa81e834d0_isrForCrosstalkSources_1125923_34.stderr parsl.dataflow.dflow INFO: Task 3347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da21ee426447f5159cda11e4a11b2186 with result from task 3347 parsl.dataflow.dflow DEBUG: Task 3347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/36d3dc80-913a-4257-b2a9-a546499da9d0_isrForCrosstalkSources_1125919_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/36d3dc80-913a-4257-b2a9-a546499da9d0_isrForCrosstalkSources_1125919_47.stderr parsl.dataflow.dflow INFO: Task 3336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc036619aa5409555f2e3554e15cc84a with result from task 3336 parsl.dataflow.dflow DEBUG: Task 3336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/7df06520-931d-4686-b88a-20846aefe82a_isrForCrosstalkSources_1125941_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/7df06520-931d-4686-b88a-20846aefe82a_isrForCrosstalkSources_1125941_26.stderr parsl.dataflow.dflow INFO: Task 3314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f26d3f3aae25aee6dbab05b58de4cb8b with result from task 3314 parsl.dataflow.dflow DEBUG: Task 3314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/f443a57a-8cc1-4802-810f-1f94dbcddbf6_isrForCrosstalkSources_1125911_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/f443a57a-8cc1-4802-810f-1f94dbcddbf6_isrForCrosstalkSources_1125911_50.stderr parsl.dataflow.dflow INFO: Task 3236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 620d5ec528dbbf5b630e1f3b11e58294 with result from task 3236 parsl.dataflow.dflow DEBUG: Task 3236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/b29922c0-df74-47bf-af3d-c2f09a3add74_isrForCrosstalkSources_1125909_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/b29922c0-df74-47bf-af3d-c2f09a3add74_isrForCrosstalkSources_1125909_3.stderr parsl.dataflow.dflow INFO: Task 3337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67fc854ed095524a3fba4e3eee213d5b with result from task 3337 parsl.dataflow.dflow DEBUG: Task 3337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a2c879f4-0837-4d31-b8b1-b588475452ca_isrForCrosstalkSources_1125933_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a2c879f4-0837-4d31-b8b1-b588475452ca_isrForCrosstalkSources_1125933_2.stderr parsl.dataflow.dflow INFO: Task 3141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd717d1b3ada0fc81f0ee0ae92c5de1e with result from task 3141 parsl.dataflow.dflow DEBUG: Task 3141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/a0369935-1af2-411e-966b-db8f7e6931ba_isrForCrosstalkSources_1125922_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/a0369935-1af2-411e-966b-db8f7e6931ba_isrForCrosstalkSources_1125922_14.stderr parsl.dataflow.dflow INFO: Task 3262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dd3ea19649db5c3a20fcba8c6b572f9 with result from task 3262 parsl.dataflow.dflow DEBUG: Task 3262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/aaf8a76f-6608-4339-8b1c-a281aedc61d4_isrForCrosstalkSources_1125941_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/aaf8a76f-6608-4339-8b1c-a281aedc61d4_isrForCrosstalkSources_1125941_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:31 proc_lsst.multi:146 [INFO] found job 16896539 in provider checkpoint proc_lsst.multi INFO: found job 16896539 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88c14e6ff354a786bb58474ab51ea957 with result from task 3317 parsl.dataflow.dflow DEBUG: Task 3317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/c5f22cd5-c88a-4c6e-9ce2-29cf427658a8_isrForCrosstalkSources_1125945_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/c5f22cd5-c88a-4c6e-9ce2-29cf427658a8_isrForCrosstalkSources_1125945_32.stderr parsl.dataflow.dflow INFO: Task 3364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 449afdc85a20ccf8efc19e76096853b5 with result from task 3364 parsl.dataflow.dflow DEBUG: Task 3364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/2a1b1e74-0831-44f1-95cc-43c8e900753a_isrForCrosstalkSources_1125928_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/2a1b1e74-0831-44f1-95cc-43c8e900753a_isrForCrosstalkSources_1125928_37.stderr parsl.dataflow.dflow INFO: Task 3204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78498ea8ab4ead14004677e3a0ff1e96 with result from task 3204 parsl.dataflow.dflow DEBUG: Task 3204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/aa5d010b-4b69-4bd6-9779-947b090cbc74_isrForCrosstalkSources_1125916_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/aa5d010b-4b69-4bd6-9779-947b090cbc74_isrForCrosstalkSources_1125916_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:31 proc_lsst.multi:146 [INFO] found job 16896540 in provider checkpoint proc_lsst.multi INFO: found job 16896540 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f22981940555b80f445627b0fa42101 with result from task 3304 parsl.dataflow.dflow DEBUG: Task 3304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/2078847b-11b5-4b2b-92b0-4694e4e2a2d2_isrForCrosstalkSources_1125931_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/2078847b-11b5-4b2b-92b0-4694e4e2a2d2_isrForCrosstalkSources_1125931_9.stderr parsl.dataflow.dflow INFO: Task 3328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac89163cb71786fa989afe6541b4e69a with result from task 3328 parsl.dataflow.dflow DEBUG: Task 3328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/4e22bed5-a71e-405d-bb0e-ee096d645966_isrForCrosstalkSources_1125896_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/4e22bed5-a71e-405d-bb0e-ee096d645966_isrForCrosstalkSources_1125896_28.stderr parsl.dataflow.dflow INFO: Task 3296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d349ca7e39afb4f65e6e8c186a5ab9fa with result from task 3296 parsl.dataflow.dflow DEBUG: Task 3296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/51758191-cedc-4ab3-aa76-42bd71e1067a_isrForCrosstalkSources_1125921_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/51758191-cedc-4ab3-aa76-42bd71e1067a_isrForCrosstalkSources_1125921_57.stderr parsl.dataflow.dflow INFO: Task 3180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d0748838b19078ba19043fd31b320f7 with result from task 3180 parsl.dataflow.dflow DEBUG: Task 3180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/b3af87a5-49f3-4ae5-b63f-3bc9904c83cd_isrForCrosstalkSources_1125916_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/b3af87a5-49f3-4ae5-b63f-3bc9904c83cd_isrForCrosstalkSources_1125916_54.stderr parsl.dataflow.dflow INFO: Task 3245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82d0bafe6d405ae852147d1af997b371 with result from task 3245 parsl.dataflow.dflow DEBUG: Task 3245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/7352328f-493f-4c7f-a4a4-784eea8a5797_isrForCrosstalkSources_1125926_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/7352328f-493f-4c7f-a4a4-784eea8a5797_isrForCrosstalkSources_1125926_18.stderr parsl.dataflow.dflow INFO: Task 3273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4ec987c43a3ec160820e5d4449faa01 with result from task 3273 parsl.dataflow.dflow DEBUG: Task 3273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/04ec1663-aa41-47c5-ae65-ced70af7fdba_isrForCrosstalkSources_1125932_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/04ec1663-aa41-47c5-ae65-ced70af7fdba_isrForCrosstalkSources_1125932_44.stderr parsl.dataflow.dflow INFO: Task 3193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4153bb8fe90d91937c438a2ba6050e6 with result from task 3193 parsl.dataflow.dflow DEBUG: Task 3193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/2c7058a4-925a-45c5-83e2-643dd2e06d03_isrForCrosstalkSources_1125920_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/2c7058a4-925a-45c5-83e2-643dd2e06d03_isrForCrosstalkSources_1125920_43.stderr parsl.dataflow.dflow INFO: Task 3292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b074f7f34728330a78f13f94936b8e0 with result from task 3292 parsl.dataflow.dflow DEBUG: Task 3292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/a0fd27ff-86b0-49c6-82b3-d37c6c0f0dc4_isrForCrosstalkSources_1125908_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/a0fd27ff-86b0-49c6-82b3-d37c6c0f0dc4_isrForCrosstalkSources_1125908_43.stderr parsl.dataflow.dflow INFO: Task 3288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c559e3ed5db63affbeb415ce0435f235 with result from task 3288 parsl.dataflow.dflow DEBUG: Task 3288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/74778877-e0ec-4a64-a402-bf612f18e8d2_isrForCrosstalkSources_1125909_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/74778877-e0ec-4a64-a402-bf612f18e8d2_isrForCrosstalkSources_1125909_34.stderr parsl.dataflow.dflow INFO: Task 3383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13a4b4880d2d4384651538f939043caa with result from task 3383 parsl.dataflow.dflow DEBUG: Task 3383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/c6fa70e2-1b6f-4cf1-8eed-e5b667519539_isrForCrosstalkSources_1125890_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/c6fa70e2-1b6f-4cf1-8eed-e5b667519539_isrForCrosstalkSources_1125890_1.stderr parsl.dataflow.dflow INFO: Task 3361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f9ea73cfb6065a70c9ef53172379289 with result from task 3361 parsl.dataflow.dflow DEBUG: Task 3361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/db004396-dec6-4381-a301-4bbc6d114088_isrForCrosstalkSources_1125926_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/db004396-dec6-4381-a301-4bbc6d114088_isrForCrosstalkSources_1125926_27.stderr parsl.dataflow.dflow INFO: Task 3299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d07d92aac19593bdb2b2597fb35d0be with result from task 3299 parsl.dataflow.dflow DEBUG: Task 3299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/b8a09bbe-5a7d-4e1b-9e15-041a2396cb25_isrForCrosstalkSources_1125891_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/b8a09bbe-5a7d-4e1b-9e15-041a2396cb25_isrForCrosstalkSources_1125891_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:32 proc_lsst.multi:146 [INFO] found job 16896541 in provider checkpoint proc_lsst.multi INFO: found job 16896541 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 653963b4f41dde81b92e049112b70421 with result from task 3365 parsl.dataflow.dflow DEBUG: Task 3365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/36260e67-f761-4c3b-90e5-631abebcfaea_isrForCrosstalkSources_1125896_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/36260e67-f761-4c3b-90e5-631abebcfaea_isrForCrosstalkSources_1125896_39.stderr parsl.dataflow.dflow INFO: Task 3217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee4bad369d1864e481ee3cb065c49486 with result from task 3217 parsl.dataflow.dflow DEBUG: Task 3217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/33162fba-68b1-4e13-967a-c53f29634ac7_isrForCrosstalkSources_1125939_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/33162fba-68b1-4e13-967a-c53f29634ac7_isrForCrosstalkSources_1125939_21.stderr parsl.dataflow.dflow INFO: Task 3306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84e5215fc472922d3f5caba25a6623e6 with result from task 3306 parsl.dataflow.dflow DEBUG: Task 3306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/c25481ac-a785-455f-a4a5-d4981b9f710b_isrForCrosstalkSources_1125940_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/c25481ac-a785-455f-a4a5-d4981b9f710b_isrForCrosstalkSources_1125940_38.stderr parsl.dataflow.dflow INFO: Task 3370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92d28894775e3caffa36eaf23e83e2ff with result from task 3370 parsl.dataflow.dflow DEBUG: Task 3370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/84a670d0-d21d-40b0-92fa-f70abae23737_isrForCrosstalkSources_1125935_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/84a670d0-d21d-40b0-92fa-f70abae23737_isrForCrosstalkSources_1125935_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:32 proc_lsst.multi:146 [INFO] found job 16896542 in provider checkpoint proc_lsst.multi INFO: found job 16896542 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee928d189263e971a28e22c26a837a07 with result from task 3179 parsl.dataflow.dflow DEBUG: Task 3179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/50ee72f6-f2d9-4c03-8001-25dcb9c5daf4_isrForCrosstalkSources_1125907_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/50ee72f6-f2d9-4c03-8001-25dcb9c5daf4_isrForCrosstalkSources_1125907_55.stderr parsl.dataflow.dflow INFO: Task 3269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08b82fe3c7752719b9d00c22e42c66b1 with result from task 3269 parsl.dataflow.dflow DEBUG: Task 3269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/e55d8f3b-d464-4fb5-99fe-82d941a5ddf8_isrForCrosstalkSources_1125893_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/e55d8f3b-d464-4fb5-99fe-82d941a5ddf8_isrForCrosstalkSources_1125893_52.stderr parsl.dataflow.dflow INFO: Task 3360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89fc7529b9705ceba97bd239a68dc2ac with result from task 3360 parsl.dataflow.dflow DEBUG: Task 3360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/9b6eaacd-42a3-4f94-a1f5-dbea09627ffc_isrForCrosstalkSources_1125942_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/9b6eaacd-42a3-4f94-a1f5-dbea09627ffc_isrForCrosstalkSources_1125942_51.stderr parsl.dataflow.dflow INFO: Task 3272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9557b91e117fa8bbac641da7e7e2d8c6 with result from task 3272 parsl.dataflow.dflow DEBUG: Task 3272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/d54975bb-27b7-4fab-b187-4463c041de54_isrForCrosstalkSources_1125920_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/d54975bb-27b7-4fab-b187-4463c041de54_isrForCrosstalkSources_1125920_25.stderr parsl.dataflow.dflow INFO: Task 3199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ddffe05f6cfa13b528d34e68f502abf with result from task 3199 parsl.dataflow.dflow DEBUG: Task 3199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/076008b4-c829-429c-9ec6-88f0a215c663_isrForCrosstalkSources_1125920_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/076008b4-c829-429c-9ec6-88f0a215c663_isrForCrosstalkSources_1125920_33.stderr parsl.dataflow.dflow INFO: Task 3388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d19873cfbbcfcda040a8dfdf4b7333e with result from task 3388 parsl.dataflow.dflow DEBUG: Task 3388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/d799f247-2a82-469d-b030-dbdd79cf47dc_isrForCrosstalkSources_1125947_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/d799f247-2a82-469d-b030-dbdd79cf47dc_isrForCrosstalkSources_1125947_39.stderr parsl.dataflow.dflow INFO: Task 3344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07c60bc68dd6d86d2958515327bc17b0 with result from task 3344 parsl.dataflow.dflow DEBUG: Task 3344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/38ec5954-8b7d-45cb-8b10-0ec4c85ace28_isrForCrosstalkSources_1125926_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/38ec5954-8b7d-45cb-8b10-0ec4c85ace28_isrForCrosstalkSources_1125926_24.stderr parsl.dataflow.dflow INFO: Task 3351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc7458520bb3ce77f6178b8d96cd24ca with result from task 3351 parsl.dataflow.dflow DEBUG: Task 3351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a9b8b3ea-9e20-461d-8a24-68d8ed0d8b91_isrForCrosstalkSources_1125938_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/a9b8b3ea-9e20-461d-8a24-68d8ed0d8b91_isrForCrosstalkSources_1125938_8.stderr parsl.dataflow.dflow INFO: Task 3330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 562565d1a39ca7915192a228aff30dc6 with result from task 3330 parsl.dataflow.dflow DEBUG: Task 3330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/7763cbbf-0627-4571-adc1-3534d280d0c0_isrForCrosstalkSources_1125926_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/7763cbbf-0627-4571-adc1-3534d280d0c0_isrForCrosstalkSources_1125926_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:32 proc_lsst.multi:146 [INFO] found job 16896543 in provider checkpoint proc_lsst.multi INFO: found job 16896543 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61df988d20be96e2a011ee1f4efe47ab with result from task 3356 parsl.dataflow.dflow DEBUG: Task 3356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3852d808-f64b-437d-bc14-04bf3c8b20c2_isrForCrosstalkSources_1125920_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/3852d808-f64b-437d-bc14-04bf3c8b20c2_isrForCrosstalkSources_1125920_49.stderr parsl.dataflow.dflow INFO: Task 3366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb3f94dd27a7940b641177ba17736c5d with result from task 3366 parsl.dataflow.dflow DEBUG: Task 3366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/048f17dd-5a91-4222-ae88-e5cb626d0bc0_isrForCrosstalkSources_1125893_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/048f17dd-5a91-4222-ae88-e5cb626d0bc0_isrForCrosstalkSources_1125893_19.stderr parsl.dataflow.dflow INFO: Task 3310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 351306e463f36585a8925d64876023e4 with result from task 3310 parsl.dataflow.dflow DEBUG: Task 3310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a6345087-82dd-4533-a7ab-4274d1c4927c_isrForCrosstalkSources_1125933_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/a6345087-82dd-4533-a7ab-4274d1c4927c_isrForCrosstalkSources_1125933_59.stderr parsl.dataflow.dflow INFO: Task 3187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4017b16578b2129787b35040901aa06d with result from task 3187 parsl.dataflow.dflow DEBUG: Task 3187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/b827aad5-981a-469b-bf12-15fe55efa624_isrForCrosstalkSources_1125901_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/b827aad5-981a-469b-bf12-15fe55efa624_isrForCrosstalkSources_1125901_47.stderr parsl.dataflow.dflow INFO: Task 3332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b984a6a0476be8d946d47de45bb8b5f with result from task 3332 parsl.dataflow.dflow DEBUG: Task 3332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/90eca44a-b19d-4061-988c-073eb2b8d947_isrForCrosstalkSources_1125935_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/90eca44a-b19d-4061-988c-073eb2b8d947_isrForCrosstalkSources_1125935_14.stderr parsl.dataflow.dflow INFO: Task 3368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71e106329d4d760786e71e61eaf26252 with result from task 3368 parsl.dataflow.dflow DEBUG: Task 3368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/53b68ef4-254f-4fc6-8f61-b87ab9e8fe8e_isrForCrosstalkSources_1125924_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/53b68ef4-254f-4fc6-8f61-b87ab9e8fe8e_isrForCrosstalkSources_1125924_3.stderr parsl.dataflow.dflow INFO: Task 3397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 650b352c402771d0e2b322e426ac87d1 with result from task 3397 parsl.dataflow.dflow DEBUG: Task 3397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/0638a0c1-5543-4c9b-a42b-afaa169cf831_isrForCrosstalkSources_1125911_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/0638a0c1-5543-4c9b-a42b-afaa169cf831_isrForCrosstalkSources_1125911_36.stderr parsl.dataflow.dflow INFO: Task 3396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2648d82d3cc3299821dc845f41ff24e5 with result from task 3396 parsl.dataflow.dflow DEBUG: Task 3396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/6f54ac21-fcd9-4f37-856a-0ff07051a337_isrForCrosstalkSources_1125920_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/6f54ac21-fcd9-4f37-856a-0ff07051a337_isrForCrosstalkSources_1125920_37.stderr parsl.dataflow.dflow INFO: Task 3374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6260dd5b8509d624f2b2e97502d9b95a with result from task 3374 parsl.dataflow.dflow DEBUG: Task 3374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/f7b72e68-e70d-4524-8ab4-6aa62542137f_isrForCrosstalkSources_1125914_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/f7b72e68-e70d-4524-8ab4-6aa62542137f_isrForCrosstalkSources_1125914_5.stderr parsl.dataflow.dflow INFO: Task 3212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81229536f94cb03241b56c112eed5d2e with result from task 3212 parsl.dataflow.dflow DEBUG: Task 3212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/95aee4d9-145f-4e2d-bc18-5c53aeac29cc_isrForCrosstalkSources_1125919_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/95aee4d9-145f-4e2d-bc18-5c53aeac29cc_isrForCrosstalkSources_1125919_53.stderr parsl.dataflow.dflow INFO: Task 3371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1694ca0c26962eeb2d39b10bb8ac1ce with result from task 3371 parsl.dataflow.dflow DEBUG: Task 3371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/23f41d5c-1fdb-4b88-8462-164cafb3c090_isrForCrosstalkSources_1125943_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/23f41d5c-1fdb-4b88-8462-164cafb3c090_isrForCrosstalkSources_1125943_43.stderr parsl.dataflow.dflow INFO: Task 3323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d81619114084925600ba6622d3c8ee82 with result from task 3323 parsl.dataflow.dflow DEBUG: Task 3323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/32785117-39b5-4a9c-aaf3-4029da2e076b_isrForCrosstalkSources_1125921_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/32785117-39b5-4a9c-aaf3-4029da2e076b_isrForCrosstalkSources_1125921_33.stderr parsl.dataflow.dflow INFO: Task 3183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 459f2d21ca6bbd6c4d3eec1e5b3f4fef with result from task 3183 parsl.dataflow.dflow DEBUG: Task 3183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/90ad9681-1774-452f-afcc-599bb51ab18a_isrForCrosstalkSources_1125893_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/90ad9681-1774-452f-afcc-599bb51ab18a_isrForCrosstalkSources_1125893_59.stderr parsl.dataflow.dflow INFO: Task 3309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 109342087f4131cc5c362d54bcfe5263 with result from task 3309 parsl.dataflow.dflow DEBUG: Task 3309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/7426e447-c3bc-4c53-82fd-b1f7f2d3b984_isrForCrosstalkSources_1125899_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/7426e447-c3bc-4c53-82fd-b1f7f2d3b984_isrForCrosstalkSources_1125899_15.stderr parsl.dataflow.dflow INFO: Task 3319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6054da85794b52d898c2743f75e197d with result from task 3319 parsl.dataflow.dflow DEBUG: Task 3319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e8fc3d35-8ece-4f41-b013-524a06106209_isrForCrosstalkSources_1125916_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/e8fc3d35-8ece-4f41-b013-524a06106209_isrForCrosstalkSources_1125916_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:33 proc_lsst.multi:146 [INFO] found job 16896544 in provider checkpoint proc_lsst.multi INFO: found job 16896544 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0258a081329bfc9beaab4e79802367a2 with result from task 3343 parsl.dataflow.dflow DEBUG: Task 3343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/6fe5fe20-3456-4a3a-afa7-50c2a7a66a9e_isrForCrosstalkSources_1125918_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/6fe5fe20-3456-4a3a-afa7-50c2a7a66a9e_isrForCrosstalkSources_1125918_20.stderr parsl.dataflow.dflow INFO: Task 3372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 707275198cfb9cdb798464b4284c1d6e with result from task 3372 parsl.dataflow.dflow DEBUG: Task 3372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/f9bf5974-484f-4a43-8052-7240e1dfca85_isrForCrosstalkSources_1125907_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/f9bf5974-484f-4a43-8052-7240e1dfca85_isrForCrosstalkSources_1125907_19.stderr parsl.dataflow.dflow INFO: Task 3412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d78d19859cd655d5f523e3bd1f6987ab with result from task 3412 parsl.dataflow.dflow DEBUG: Task 3412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c1b37da5-c009-4158-9408-f6e6e3949244_isrForCrosstalkSources_1125916_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c1b37da5-c009-4158-9408-f6e6e3949244_isrForCrosstalkSources_1125916_7.stderr parsl.dataflow.dflow INFO: Task 3320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c906df3018c67876f2ff5dad2d89f366 with result from task 3320 parsl.dataflow.dflow DEBUG: Task 3320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/d3e4f0e9-dcfa-4b60-b45d-6d35139ce119_isrForCrosstalkSources_1125899_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/d3e4f0e9-dcfa-4b60-b45d-6d35139ce119_isrForCrosstalkSources_1125899_43.stderr parsl.dataflow.dflow INFO: Task 3277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5d48c0adadf381c52f7755624b40bce with result from task 3277 parsl.dataflow.dflow DEBUG: Task 3277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/fba49205-95a7-41ac-bd26-11b9d9326e54_isrForCrosstalkSources_1125945_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/fba49205-95a7-41ac-bd26-11b9d9326e54_isrForCrosstalkSources_1125945_55.stderr parsl.dataflow.dflow INFO: Task 3375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6401b36d3b03a85c7feb5e7b6509e413 with result from task 3375 parsl.dataflow.dflow DEBUG: Task 3375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/329d093f-04a7-494f-93d8-be2966069258_isrForCrosstalkSources_1125893_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/329d093f-04a7-494f-93d8-be2966069258_isrForCrosstalkSources_1125893_58.stderr parsl.dataflow.dflow INFO: Task 3338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f96e020d72e21bcdafd56fb36f240b3e with result from task 3338 parsl.dataflow.dflow DEBUG: Task 3338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/cd9f2a79-9a23-469d-945c-c5541e3e4765_isrForCrosstalkSources_1125935_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/cd9f2a79-9a23-469d-945c-c5541e3e4765_isrForCrosstalkSources_1125935_15.stderr parsl.dataflow.dflow INFO: Task 3301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e6d79aed075200a801d1f8a93dee315 with result from task 3301 parsl.dataflow.dflow DEBUG: Task 3301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/5c16359e-bfac-452c-b2e9-e1cb79ac92b7_isrForCrosstalkSources_1125916_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/5c16359e-bfac-452c-b2e9-e1cb79ac92b7_isrForCrosstalkSources_1125916_53.stderr parsl.dataflow.dflow INFO: Task 3233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df9936cc32f1ede9aa305a4150a5bb38 with result from task 3233 parsl.dataflow.dflow DEBUG: Task 3233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/029f4c92-1cb2-4b53-be97-806c6c6c20f5_isrForCrosstalkSources_1125906_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/029f4c92-1cb2-4b53-be97-806c6c6c20f5_isrForCrosstalkSources_1125906_29.stderr parsl.dataflow.dflow INFO: Task 3416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d2aab99c0fb91d3430430345f4a0182 with result from task 3416 parsl.dataflow.dflow DEBUG: Task 3416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/c0fba0ae-5cf5-4da5-ba0f-2bfbbe060fe1_isrForCrosstalkSources_1125938_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/c0fba0ae-5cf5-4da5-ba0f-2bfbbe060fe1_isrForCrosstalkSources_1125938_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:33 proc_lsst.multi:146 [INFO] found job 16896545 in provider checkpoint proc_lsst.multi INFO: found job 16896545 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 892ce461820f4b0a1916351a6a370d1b with result from task 3427 parsl.dataflow.dflow DEBUG: Task 3427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f5057a27-d34a-4a89-89fd-3521fce168ce_isrForCrosstalkSources_1125922_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f5057a27-d34a-4a89-89fd-3521fce168ce_isrForCrosstalkSources_1125922_20.stderr parsl.dataflow.dflow INFO: Task 3329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2e633b97fadc180fe9d7d81acba1d66 with result from task 3329 parsl.dataflow.dflow DEBUG: Task 3329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/9845aac7-1156-46a1-87ec-f1e7367a6738_isrForCrosstalkSources_1125938_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/9845aac7-1156-46a1-87ec-f1e7367a6738_isrForCrosstalkSources_1125938_15.stderr parsl.dataflow.dflow INFO: Task 3411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 759a26eb619176fddb727be566299709 with result from task 3411 parsl.dataflow.dflow DEBUG: Task 3411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/9ebac8d9-dfce-4ab7-9ff0-92231ca0074c_isrForCrosstalkSources_1125915_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/9ebac8d9-dfce-4ab7-9ff0-92231ca0074c_isrForCrosstalkSources_1125915_42.stderr parsl.dataflow.dflow INFO: Task 3381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07695cec289a300527cf1384cd158fc1 with result from task 3381 parsl.dataflow.dflow DEBUG: Task 3381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/884389d2-329a-4a62-83ad-708fd93949be_isrForCrosstalkSources_1125910_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/884389d2-329a-4a62-83ad-708fd93949be_isrForCrosstalkSources_1125910_46.stderr parsl.dataflow.dflow INFO: Task 3352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea45dd870315fb9e80a39fc8c2294d50 with result from task 3352 parsl.dataflow.dflow DEBUG: Task 3352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/648cbc32-be57-43b0-a205-13c652989f12_isrForCrosstalkSources_1125915_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/648cbc32-be57-43b0-a205-13c652989f12_isrForCrosstalkSources_1125915_17.stderr parsl.dataflow.dflow INFO: Task 3208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd74d07176dd0ee60fbf4cfef1b33635 with result from task 3208 parsl.dataflow.dflow DEBUG: Task 3208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/d2831846-4a2b-48ea-be77-60f48fb193ce_isrForCrosstalkSources_1125925_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/d2831846-4a2b-48ea-be77-60f48fb193ce_isrForCrosstalkSources_1125925_53.stderr parsl.dataflow.dflow INFO: Task 3307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d68ca9884d733f3b45441ca3d60488a with result from task 3307 parsl.dataflow.dflow DEBUG: Task 3307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/d3aef5d9-bf0b-46e1-8e6a-37fea01fa4f7_isrForCrosstalkSources_1125920_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/d3aef5d9-bf0b-46e1-8e6a-37fea01fa4f7_isrForCrosstalkSources_1125920_61.stderr parsl.dataflow.dflow INFO: Task 3355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 856c5f03e2ad52612d178edb1e98bc53 with result from task 3355 parsl.dataflow.dflow DEBUG: Task 3355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/61f21116-753f-4327-8c45-5a0349bf2367_isrForCrosstalkSources_1125935_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/61f21116-753f-4327-8c45-5a0349bf2367_isrForCrosstalkSources_1125935_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:34 proc_lsst.multi:146 [INFO] found job 16896546 in provider checkpoint proc_lsst.multi INFO: found job 16896546 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 322677e58598b0cdb271c7b2151d9bd1 with result from task 3318 parsl.dataflow.dflow DEBUG: Task 3318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/5a810ae5-516f-4cc1-bc4a-2f2d6a7e162a_isrForCrosstalkSources_1125888_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/5a810ae5-516f-4cc1-bc4a-2f2d6a7e162a_isrForCrosstalkSources_1125888_1.stderr parsl.dataflow.dflow INFO: Task 3380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d986a9c733fc05b63f1904027a2f7d4 with result from task 3380 parsl.dataflow.dflow DEBUG: Task 3380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/ef2a79fc-a87f-486a-ac24-99ae040db962_isrForCrosstalkSources_1125938_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/ef2a79fc-a87f-486a-ac24-99ae040db962_isrForCrosstalkSources_1125938_12.stderr parsl.dataflow.dflow INFO: Task 3424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc87655f6f6a3a366bb3a22ebee3da46 with result from task 3424 parsl.dataflow.dflow DEBUG: Task 3424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/3e7685ea-5ea4-4560-b669-ae9f593a9238_isrForCrosstalkSources_1125924_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/3e7685ea-5ea4-4560-b669-ae9f593a9238_isrForCrosstalkSources_1125924_30.stderr parsl.dataflow.dflow INFO: Task 3170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e195a147064e7b6d11d8fde7c562fa61 with result from task 3170 parsl.dataflow.dflow DEBUG: Task 3170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/8e379050-1148-4c8b-a2be-dbb6a0e3368b_isrForCrosstalkSources_1125888_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/8e379050-1148-4c8b-a2be-dbb6a0e3368b_isrForCrosstalkSources_1125888_54.stderr parsl.dataflow.dflow INFO: Task 3280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dcc5ad4f0ead103532f8ad200ffb996 with result from task 3280 parsl.dataflow.dflow DEBUG: Task 3280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/6a280309-5d57-42b8-9458-0420c78fa624_isrForCrosstalkSources_1125905_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/6a280309-5d57-42b8-9458-0420c78fa624_isrForCrosstalkSources_1125905_31.stderr parsl.dataflow.dflow INFO: Task 3281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c34aebbafaf883f3c7a79d38e22bf8d with result from task 3281 parsl.dataflow.dflow DEBUG: Task 3281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/7f257e46-86af-48a3-bcb1-30ab6a61d6c7_isrForCrosstalkSources_1125917_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/7f257e46-86af-48a3-bcb1-30ab6a61d6c7_isrForCrosstalkSources_1125917_8.stderr parsl.dataflow.dflow INFO: Task 3394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 219f1ad30bd1a52aee0da3c3ee05791a with result from task 3394 parsl.dataflow.dflow DEBUG: Task 3394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a323afe7-d47e-4d1c-92b2-a7b5415233fb_isrForCrosstalkSources_1125909_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/a323afe7-d47e-4d1c-92b2-a7b5415233fb_isrForCrosstalkSources_1125909_18.stderr parsl.dataflow.dflow INFO: Task 3302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 554b24dd6b2a8ed8e2452cd52e9c67b4 with result from task 3302 parsl.dataflow.dflow DEBUG: Task 3302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/c3b98ec2-2695-4840-b1c5-2edc477919fe_isrForCrosstalkSources_1125912_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/c3b98ec2-2695-4840-b1c5-2edc477919fe_isrForCrosstalkSources_1125912_27.stderr parsl.dataflow.dflow INFO: Task 3143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63ef0e59d7888921ea8a255405caf1ac with result from task 3143 parsl.dataflow.dflow DEBUG: Task 3143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/a8143ab0-eb00-42e4-8a08-01f148c94388_isrForCrosstalkSources_1125926_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/a8143ab0-eb00-42e4-8a08-01f148c94388_isrForCrosstalkSources_1125926_10.stderr parsl.dataflow.dflow INFO: Task 3331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b006955b39497bd458cf12f1ff4b9bd3 with result from task 3331 parsl.dataflow.dflow DEBUG: Task 3331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/42e64dae-a06a-48bb-be56-44b14c20f686_isrForCrosstalkSources_1125897_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/42e64dae-a06a-48bb-be56-44b14c20f686_isrForCrosstalkSources_1125897_8.stderr parsl.dataflow.dflow INFO: Task 3255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abd4a14698bd3634233c9d86409f26b1 with result from task 3255 parsl.dataflow.dflow DEBUG: Task 3255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/b50bc973-cba0-419f-a51b-cddaff2ace9b_isrForCrosstalkSources_1125946_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/b50bc973-cba0-419f-a51b-cddaff2ace9b_isrForCrosstalkSources_1125946_58.stderr parsl.dataflow.dflow INFO: Task 3385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 355278652aa2781402496b112a0edda0 with result from task 3385 parsl.dataflow.dflow DEBUG: Task 3385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/f482997d-6d8a-48ce-8a22-fc150abd564b_isrForCrosstalkSources_1125938_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/f482997d-6d8a-48ce-8a22-fc150abd564b_isrForCrosstalkSources_1125938_51.stderr parsl.dataflow.dflow INFO: Task 3376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 288156ed677292e23e0c49bb78e9938c with result from task 3376 parsl.dataflow.dflow DEBUG: Task 3376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/d2c5c3fd-b48d-4768-be1e-7bd6eedc416b_isrForCrosstalkSources_1125929_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/d2c5c3fd-b48d-4768-be1e-7bd6eedc416b_isrForCrosstalkSources_1125929_56.stderr parsl.dataflow.dflow INFO: Task 3393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ba7ddf8bb70c136056b2a317aa20196 with result from task 3393 parsl.dataflow.dflow DEBUG: Task 3393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/d34f27db-d9d9-446a-a6ec-29863863f97d_isrForCrosstalkSources_1125936_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/d34f27db-d9d9-446a-a6ec-29863863f97d_isrForCrosstalkSources_1125936_39.stderr parsl.dataflow.dflow INFO: Task 3248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 074f6fc4d623ed24223bb96c37b12705 with result from task 3248 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:34 proc_lsst.multi:146 [INFO] found job 16896547 in provider checkpoint proc_lsst.multi INFO: found job 16896547 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 3248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/806e9755-5394-41d7-b7fd-58d15d34ccd3_isrForCrosstalkSources_1125905_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/806e9755-5394-41d7-b7fd-58d15d34ccd3_isrForCrosstalkSources_1125905_2.stderr parsl.dataflow.dflow INFO: Task 3261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a8934ad51b5523ee09b22b165d39c70 with result from task 3261 parsl.dataflow.dflow DEBUG: Task 3261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/8acbdf7a-ea03-4c3b-9316-8b52cfe639cc_isrForCrosstalkSources_1125926_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/8acbdf7a-ea03-4c3b-9316-8b52cfe639cc_isrForCrosstalkSources_1125926_52.stderr parsl.dataflow.dflow INFO: Task 3324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e03925cafe7d5875617f2eb1842b1578 with result from task 3324 parsl.dataflow.dflow DEBUG: Task 3324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/5bcaa6fa-d146-4f8b-8968-f8dbed0f7feb_isrForCrosstalkSources_1125946_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/5bcaa6fa-d146-4f8b-8968-f8dbed0f7feb_isrForCrosstalkSources_1125946_2.stderr parsl.dataflow.dflow INFO: Task 3214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 839a45282c5cea98d351c59904b23a4a with result from task 3214 parsl.dataflow.dflow DEBUG: Task 3214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/1a5754ac-8687-4531-9399-5d47c40551cf_isrForCrosstalkSources_1125923_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/1a5754ac-8687-4531-9399-5d47c40551cf_isrForCrosstalkSources_1125923_22.stderr parsl.dataflow.dflow INFO: Task 3349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17738a4881976039da0860511df867e3 with result from task 3349 parsl.dataflow.dflow DEBUG: Task 3349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/4a130c25-d530-4d62-afc6-9d6ed7125179_isrForCrosstalkSources_1125915_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/4a130c25-d530-4d62-afc6-9d6ed7125179_isrForCrosstalkSources_1125915_57.stderr parsl.dataflow.dflow INFO: Task 3334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2007a08db30168a603c196a6a3fc1470 with result from task 3334 parsl.dataflow.dflow DEBUG: Task 3334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/22f4aeda-60f9-449f-9a4c-718f59f3f776_isrForCrosstalkSources_1125912_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/22f4aeda-60f9-449f-9a4c-718f59f3f776_isrForCrosstalkSources_1125912_2.stderr parsl.dataflow.dflow INFO: Task 3407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c1268485f415d0942d1fb6f825dc7be with result from task 3407 parsl.dataflow.dflow DEBUG: Task 3407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/ab8eab0e-b8e9-4427-a126-9508c3573bb8_isrForCrosstalkSources_1125905_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/ab8eab0e-b8e9-4427-a126-9508c3573bb8_isrForCrosstalkSources_1125905_17.stderr parsl.dataflow.dflow INFO: Task 3379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29373feb203f0b780441813da2a136dd with result from task 3379 parsl.dataflow.dflow DEBUG: Task 3379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2a31e8b6-499c-4566-9d87-1593677471b0_isrForCrosstalkSources_1125933_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2a31e8b6-499c-4566-9d87-1593677471b0_isrForCrosstalkSources_1125933_30.stderr parsl.dataflow.dflow INFO: Task 3357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3563a0d9bec8228d610fb9dd08ab0e08 with result from task 3357 parsl.dataflow.dflow DEBUG: Task 3357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a4aa1150-569d-47a3-b90a-c59be1327f78_isrForCrosstalkSources_1125921_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a4aa1150-569d-47a3-b90a-c59be1327f78_isrForCrosstalkSources_1125921_2.stderr parsl.dataflow.dflow INFO: Task 3377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd9fc261c0664989289e9e368fe2022c with result from task 3377 parsl.dataflow.dflow DEBUG: Task 3377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/468d8e8b-0d45-4d6b-8f54-d2ca29e7b17d_isrForCrosstalkSources_1125926_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/468d8e8b-0d45-4d6b-8f54-d2ca29e7b17d_isrForCrosstalkSources_1125926_45.stderr parsl.dataflow.dflow INFO: Task 3369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fff33a3392f1e5e695bd870d508eacdd with result from task 3369 parsl.dataflow.dflow DEBUG: Task 3369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/07cc0e37-3749-428d-8ed8-c28d89fe286c_isrForCrosstalkSources_1125928_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/07cc0e37-3749-428d-8ed8-c28d89fe286c_isrForCrosstalkSources_1125928_9.stderr parsl.dataflow.dflow INFO: Task 3373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab4c3b13e3fa3c458a93fb5d481fcba4 with result from task 3373 parsl.dataflow.dflow DEBUG: Task 3373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/b05fbf42-bae0-4225-889e-a7deaa8f8b89_isrForCrosstalkSources_1125914_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/b05fbf42-bae0-4225-889e-a7deaa8f8b89_isrForCrosstalkSources_1125914_30.stderr parsl.dataflow.dflow INFO: Task 3184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9be409203ba0465a1471755c60a94e2d with result from task 3184 parsl.dataflow.dflow DEBUG: Task 3184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/1b4a3194-f411-4514-ad25-e3b41deaf281_isrForCrosstalkSources_1125919_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/1b4a3194-f411-4514-ad25-e3b41deaf281_isrForCrosstalkSources_1125919_49.stderr parsl.dataflow.dflow INFO: Task 3425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e8276717368eb4023d4146c33421fc6 with result from task 3425 parsl.dataflow.dflow DEBUG: Task 3425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/a4569d9e-3e7b-41ea-b72a-b60c17dc2dd4_isrForCrosstalkSources_1125912_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/a4569d9e-3e7b-41ea-b72a-b60c17dc2dd4_isrForCrosstalkSources_1125912_53.stderr parsl.dataflow.dflow INFO: Task 3156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7322b9dbd17afafdb4ed112f084593a0 with result from task 3156 parsl.dataflow.dflow DEBUG: Task 3156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/1a5db8f0-9f63-45b4-8996-32ba332aad47_isrForCrosstalkSources_1125919_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/1a5db8f0-9f63-45b4-8996-32ba332aad47_isrForCrosstalkSources_1125919_12.stderr parsl.dataflow.dflow INFO: Task 3247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 170bfa68ab2d2a229afbfaa1955e24f4 with result from task 3247 parsl.dataflow.dflow DEBUG: Task 3247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/129919c0-00f2-48b1-b85b-0ad8366b161e_isrForCrosstalkSources_1125910_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/129919c0-00f2-48b1-b85b-0ad8366b161e_isrForCrosstalkSources_1125910_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:35 proc_lsst.multi:146 [INFO] found job 16896548 in provider checkpoint proc_lsst.multi INFO: found job 16896548 in provider checkpoint parsl.dataflow.dflow INFO: Task 3227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78bc22eedcb0c8f8a5304689afd7b438 with result from task 3227 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 3227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e4cf8b8c-9015-45b3-883a-2baeb179cdb3_isrForCrosstalkSources_1125942_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/e4cf8b8c-9015-45b3-883a-2baeb179cdb3_isrForCrosstalkSources_1125942_49.stderr parsl.dataflow.dflow INFO: Task 3353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e6bb232a5e4bde232e01cdc67307e87 with result from task 3353 parsl.dataflow.dflow DEBUG: Task 3353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/6eb105f4-d91e-4e40-bc4b-2b4583228a94_isrForCrosstalkSources_1125906_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/6eb105f4-d91e-4e40-bc4b-2b4583228a94_isrForCrosstalkSources_1125906_49.stderr parsl.dataflow.dflow INFO: Task 3283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54d39a33ec081cb08d342c3fd77584bd with result from task 3283 parsl.dataflow.dflow DEBUG: Task 3283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3ec6298a-bf5f-4560-80b3-07e321e2f0f8_isrForCrosstalkSources_1125941_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3ec6298a-bf5f-4560-80b3-07e321e2f0f8_isrForCrosstalkSources_1125941_6.stderr parsl.dataflow.dflow INFO: Task 3148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f05462fd1e0ac8236846f436d0f18c01 with result from task 3148 parsl.dataflow.dflow DEBUG: Task 3148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/3c752457-4fbd-4539-9a62-0f0f9e391107_isrForCrosstalkSources_1125946_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/3c752457-4fbd-4539-9a62-0f0f9e391107_isrForCrosstalkSources_1125946_18.stderr parsl.dataflow.dflow INFO: Task 3275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 219d1b93f750bbd0a32e38eb964145f6 with result from task 3275 parsl.dataflow.dflow DEBUG: Task 3275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/c30679c6-5e32-48ce-ba3a-ef88467b5547_isrForCrosstalkSources_1125934_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/c30679c6-5e32-48ce-ba3a-ef88467b5547_isrForCrosstalkSources_1125934_10.stderr parsl.dataflow.dflow INFO: Task 3237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af318365b4fd4239725cfcaba072f2a3 with result from task 3237 parsl.dataflow.dflow DEBUG: Task 3237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/bcfdcca6-ecc6-4ec4-9c32-3d3e4ed8454d_isrForCrosstalkSources_1125902_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/bcfdcca6-ecc6-4ec4-9c32-3d3e4ed8454d_isrForCrosstalkSources_1125902_53.stderr parsl.dataflow.dflow INFO: Task 3409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82a6f46aa0149220ea2fd776eedf950f with result from task 3409 parsl.dataflow.dflow DEBUG: Task 3409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/3a1b0051-4859-4be8-b348-4ca805043dd2_isrForCrosstalkSources_1125924_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/3a1b0051-4859-4be8-b348-4ca805043dd2_isrForCrosstalkSources_1125924_26.stderr parsl.dataflow.dflow INFO: Task 3291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1799c9426ae156fccd0fd7ba18edfef5 with result from task 3291 parsl.dataflow.dflow DEBUG: Task 3291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/1e15539c-cfa4-45c7-bf68-063fe0af6dd7_isrForCrosstalkSources_1125892_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/1e15539c-cfa4-45c7-bf68-063fe0af6dd7_isrForCrosstalkSources_1125892_59.stderr parsl.dataflow.dflow INFO: Task 3286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0701ce9100d67641ace1a28449d07c4a with result from task 3286 parsl.dataflow.dflow DEBUG: Task 3286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/37ab7dc9-e7ec-41da-a85e-3ab00d658d2a_isrForCrosstalkSources_1125929_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/37ab7dc9-e7ec-41da-a85e-3ab00d658d2a_isrForCrosstalkSources_1125929_53.stderr parsl.dataflow.dflow INFO: Task 3405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 502ca86e43d96bed9851ecca87025519 with result from task 3405 parsl.dataflow.dflow DEBUG: Task 3405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/fa6046bc-e06e-484e-894d-41926ce31a6d_isrForCrosstalkSources_1125935_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/fa6046bc-e06e-484e-894d-41926ce31a6d_isrForCrosstalkSources_1125935_20.stderr parsl.dataflow.dflow INFO: Task 3305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d8968eaf1d4759cb12368856fb94e22 with result from task 3305 parsl.dataflow.dflow DEBUG: Task 3305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/41a90ab0-6aba-48f0-a153-1b48a6f594a2_isrForCrosstalkSources_1125906_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/41a90ab0-6aba-48f0-a153-1b48a6f594a2_isrForCrosstalkSources_1125906_35.stderr parsl.dataflow.dflow INFO: Task 3123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc81f36def681a4cb54852a23bc1e397 with result from task 3123 parsl.dataflow.dflow DEBUG: Task 3123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/c0339678-fa18-47cb-a785-8c0d838b5c19_isrForCrosstalkSources_1125935_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/c0339678-fa18-47cb-a785-8c0d838b5c19_isrForCrosstalkSources_1125935_6.stderr parsl.dataflow.dflow INFO: Task 3293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab48273b986c69d080e0202ddbff8b49 with result from task 3293 parsl.dataflow.dflow DEBUG: Task 3293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/e34ab16e-a183-4627-ab5f-c952d4ea0aab_isrForCrosstalkSources_1125938_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/e34ab16e-a183-4627-ab5f-c952d4ea0aab_isrForCrosstalkSources_1125938_2.stderr parsl.dataflow.dflow INFO: Task 3173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32bc4b67ebe87f2431ef5d30a6116964 with result from task 3173 parsl.dataflow.dflow DEBUG: Task 3173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/98e22b85-600b-4628-866b-8eb7176d49a4_isrForCrosstalkSources_1125901_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/98e22b85-600b-4628-866b-8eb7176d49a4_isrForCrosstalkSources_1125901_14.stderr parsl.dataflow.dflow INFO: Task 3312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2db270f436b8e63da905d86653a30719 with result from task 3312 parsl.dataflow.dflow DEBUG: Task 3312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6f87bc43-bc5d-470f-a650-ad476b0e63ac_isrForCrosstalkSources_1125927_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6f87bc43-bc5d-470f-a650-ad476b0e63ac_isrForCrosstalkSources_1125927_50.stderr parsl.dataflow.dflow INFO: Task 3294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e34838800794f2061c6ca5b4fa03b1d5 with result from task 3294 parsl.dataflow.dflow DEBUG: Task 3294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/178c4800-8039-4107-aa47-4d3988cc13df_isrForCrosstalkSources_1125921_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/178c4800-8039-4107-aa47-4d3988cc13df_isrForCrosstalkSources_1125921_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:35 proc_lsst.multi:146 [INFO] found job 16896549 in provider checkpoint proc_lsst.multi INFO: found job 16896549 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd9bcbe4844be0e95642bef03c2deeca with result from task 3422 parsl.dataflow.dflow DEBUG: Task 3422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6cb2de9f-e6c5-497e-8674-1c7e9b47713a_isrForCrosstalkSources_1125927_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6cb2de9f-e6c5-497e-8674-1c7e9b47713a_isrForCrosstalkSources_1125927_4.stderr parsl.dataflow.dflow INFO: Task 3169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 039e7bc809a9acb18828085ae7de5fef with result from task 3169 parsl.dataflow.dflow DEBUG: Task 3169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/7548874f-429e-4b52-87dc-ea3409b9c059_isrForCrosstalkSources_1125903_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/7548874f-429e-4b52-87dc-ea3409b9c059_isrForCrosstalkSources_1125903_6.stderr parsl.dataflow.dflow INFO: Task 3253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa953bc81be3f853fec7cd078a753305 with result from task 3253 parsl.dataflow.dflow DEBUG: Task 3253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ae548a9f-a255-418d-86eb-49dd03814f2b_isrForCrosstalkSources_1125918_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/ae548a9f-a255-418d-86eb-49dd03814f2b_isrForCrosstalkSources_1125918_23.stderr parsl.dataflow.dflow INFO: Task 3367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02bf6420973e6a1a40fb3ab38788f06f with result from task 3367 parsl.dataflow.dflow DEBUG: Task 3367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/146ef217-2f11-4b2d-a2e4-7bd1600f4365_isrForCrosstalkSources_1125933_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/146ef217-2f11-4b2d-a2e4-7bd1600f4365_isrForCrosstalkSources_1125933_53.stderr parsl.dataflow.dflow INFO: Task 3384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe2d6b9cb85db4140b2b8b434d7e9fca with result from task 3384 parsl.dataflow.dflow DEBUG: Task 3384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/8c624f4b-9760-4568-8f7c-fb423ada9194_isrForCrosstalkSources_1125907_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/8c624f4b-9760-4568-8f7c-fb423ada9194_isrForCrosstalkSources_1125907_28.stderr parsl.dataflow.dflow INFO: Task 3446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 306665c9cf4707e8ebbacdc70c41a154 with result from task 3446 parsl.dataflow.dflow DEBUG: Task 3446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/221f3eeb-1fcc-467b-b825-028232e06994_isrForCrosstalkSources_1125922_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/221f3eeb-1fcc-467b-b825-028232e06994_isrForCrosstalkSources_1125922_44.stderr parsl.dataflow.dflow INFO: Task 3335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ada104746242e25500ae5136634e6f6 with result from task 3335 parsl.dataflow.dflow DEBUG: Task 3335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/96099489-d02e-4e1f-b8ec-13be595fab5b_isrForCrosstalkSources_1125901_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/96099489-d02e-4e1f-b8ec-13be595fab5b_isrForCrosstalkSources_1125901_29.stderr parsl.dataflow.dflow INFO: Task 3358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 210904dc47d9dcd24ff5b0e1bebfb7f6 with result from task 3358 parsl.dataflow.dflow DEBUG: Task 3358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/0c9e94d3-c97d-4ffb-97e0-b966078f7ce7_isrForCrosstalkSources_1125920_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/0c9e94d3-c97d-4ffb-97e0-b966078f7ce7_isrForCrosstalkSources_1125920_42.stderr parsl.dataflow.dflow INFO: Task 3327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a6e654cab9e080c222f3a3ca701e8ef with result from task 3327 parsl.dataflow.dflow DEBUG: Task 3327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/b11f8f93-7dc9-4c4c-9e94-318a01fdf9d6_isrForCrosstalkSources_1125907_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/b11f8f93-7dc9-4c4c-9e94-318a01fdf9d6_isrForCrosstalkSources_1125907_58.stderr parsl.dataflow.dflow INFO: Task 3186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a7aae04166ae3305e57ceeb303924fd with result from task 3186 parsl.dataflow.dflow DEBUG: Task 3186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/17f28ba7-c350-4b88-b90b-a66a5545bee2_isrForCrosstalkSources_1125896_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/17f28ba7-c350-4b88-b90b-a66a5545bee2_isrForCrosstalkSources_1125896_54.stderr parsl.dataflow.dflow INFO: Task 3230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e258053d3161118a83a3fd9bc2ca2f9 with result from task 3230 parsl.dataflow.dflow DEBUG: Task 3230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/02af419d-75c7-453d-bf1d-b966504fd19d_isrForCrosstalkSources_1125891_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/02af419d-75c7-453d-bf1d-b966504fd19d_isrForCrosstalkSources_1125891_24.stderr parsl.dataflow.dflow INFO: Task 3211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry def70b29989b227e79d15e1e847b1cf7 with result from task 3211 parsl.dataflow.dflow DEBUG: Task 3211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/185286ee-bb96-4995-8efd-ee3f41d32eab_isrForCrosstalkSources_1125926_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/185286ee-bb96-4995-8efd-ee3f41d32eab_isrForCrosstalkSources_1125926_3.stderr parsl.dataflow.dflow INFO: Task 3298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ec30841897d8beb4c3d8faf0433de35 with result from task 3298 parsl.dataflow.dflow DEBUG: Task 3298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/61c48912-562d-48a0-bb61-866aac27570e_isrForCrosstalkSources_1125928_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/61c48912-562d-48a0-bb61-866aac27570e_isrForCrosstalkSources_1125928_62.stderr parsl.dataflow.dflow INFO: Task 3297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a828d8714dd242ad6da8ef65ec9731c with result from task 3297 parsl.dataflow.dflow DEBUG: Task 3297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/cd04e677-0bd8-46b9-821f-c53078e1a62d_isrForCrosstalkSources_1125909_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/cd04e677-0bd8-46b9-821f-c53078e1a62d_isrForCrosstalkSources_1125909_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:36 proc_lsst.multi:146 [INFO] found job 16896550 in provider checkpoint proc_lsst.multi INFO: found job 16896550 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b147984ef1a7873431348bff7018071c with result from task 3426 parsl.dataflow.dflow DEBUG: Task 3426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/4b0f5b93-a29c-49e4-95b1-827b09f2ce1c_isrForCrosstalkSources_1125911_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/4b0f5b93-a29c-49e4-95b1-827b09f2ce1c_isrForCrosstalkSources_1125911_27.stderr parsl.dataflow.dflow INFO: Task 3429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbd8575a03eb3bae4c2e7401a2e620cc with result from task 3429 parsl.dataflow.dflow DEBUG: Task 3429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d6934c01-976b-455a-a854-9d3b0002f24f_isrForCrosstalkSources_1125915_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d6934c01-976b-455a-a854-9d3b0002f24f_isrForCrosstalkSources_1125915_31.stderr parsl.dataflow.dflow INFO: Task 3209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e767fce6d302d21ebd3a23ade34acb59 with result from task 3209 parsl.dataflow.dflow DEBUG: Task 3209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/1b03f3d4-6e64-4da0-b33e-f9d6c6068e50_isrForCrosstalkSources_1125897_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/1b03f3d4-6e64-4da0-b33e-f9d6c6068e50_isrForCrosstalkSources_1125897_60.stderr parsl.dataflow.dflow INFO: Task 3410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6917c3f9b2342f315ae14313a53ce18d with result from task 3410 parsl.dataflow.dflow DEBUG: Task 3410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/304307af-506b-46a4-bee6-850c6fb847fb_isrForCrosstalkSources_1125946_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/304307af-506b-46a4-bee6-850c6fb847fb_isrForCrosstalkSources_1125946_52.stderr parsl.dataflow.dflow INFO: Task 3430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25989f927fa0cb9dcad5050cb0d478e8 with result from task 3430 parsl.dataflow.dflow DEBUG: Task 3430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/e302a956-e170-4586-a124-3fbf110330ed_isrForCrosstalkSources_1125912_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/e302a956-e170-4586-a124-3fbf110330ed_isrForCrosstalkSources_1125912_13.stderr parsl.dataflow.dflow INFO: Task 3392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f9fb023dd371df20c0c31c0490b499d with result from task 3392 parsl.dataflow.dflow DEBUG: Task 3392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/31ab40e5-400a-4792-b2c0-ac5591ee4064_isrForCrosstalkSources_1125922_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/31ab40e5-400a-4792-b2c0-ac5591ee4064_isrForCrosstalkSources_1125922_62.stderr parsl.dataflow.dflow INFO: Task 3415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4eb62d039aac53fb80b64eae0384ffe9 with result from task 3415 parsl.dataflow.dflow DEBUG: Task 3415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/38117664-4ef6-4223-8886-764613d7ec14_isrForCrosstalkSources_1125889_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/38117664-4ef6-4223-8886-764613d7ec14_isrForCrosstalkSources_1125889_28.stderr parsl.dataflow.dflow INFO: Task 3432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fde53d28bf582af139720fe40479ed0 with result from task 3432 parsl.dataflow.dflow DEBUG: Task 3432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/7d806a74-b973-485a-83fe-d0a20f3afa6a_isrForCrosstalkSources_1125933_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/7d806a74-b973-485a-83fe-d0a20f3afa6a_isrForCrosstalkSources_1125933_4.stderr parsl.dataflow.dflow INFO: Task 3454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bd8db466678f612acae07f9a7d36e63 with result from task 3454 parsl.dataflow.dflow DEBUG: Task 3454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/1af2a2e6-6a5e-444b-8502-04e6b00a3b27_isrForCrosstalkSources_1125936_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/1af2a2e6-6a5e-444b-8502-04e6b00a3b27_isrForCrosstalkSources_1125936_51.stderr parsl.dataflow.dflow INFO: Task 3440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f20ffa0861242b5bafc9732747dc8e6 with result from task 3440 parsl.dataflow.dflow DEBUG: Task 3440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/29d0a34e-3859-4516-809e-9c0de56bb4af_isrForCrosstalkSources_1125902_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/29d0a34e-3859-4516-809e-9c0de56bb4af_isrForCrosstalkSources_1125902_20.stderr parsl.dataflow.dflow INFO: Task 3205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 658f1f2cf84106a58aaf00fe5cf766e6 with result from task 3205 parsl.dataflow.dflow DEBUG: Task 3205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/9ff4a26a-ebf2-4695-a4df-2105d4a7467f_isrForCrosstalkSources_1125947_61.stdout parsl.dataflow.dflow INFO: Standard error for task 3205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/9ff4a26a-ebf2-4695-a4df-2105d4a7467f_isrForCrosstalkSources_1125947_61.stderr parsl.dataflow.dflow INFO: Task 3399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e04a3c5904b019a427f748c00156a66 with result from task 3399 parsl.dataflow.dflow DEBUG: Task 3399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/c4f06387-9265-405d-97c6-46dda499d9f6_isrForCrosstalkSources_1125934_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/c4f06387-9265-405d-97c6-46dda499d9f6_isrForCrosstalkSources_1125934_48.stderr parsl.dataflow.dflow INFO: Task 3436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53a7c15cdb599672f36a960e1a7e5005 with result from task 3436 parsl.dataflow.dflow DEBUG: Task 3436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/9bc4cb2f-1271-4efb-acc8-05aeb74ca560_isrForCrosstalkSources_1125939_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/9bc4cb2f-1271-4efb-acc8-05aeb74ca560_isrForCrosstalkSources_1125939_32.stderr parsl.dataflow.dflow INFO: Task 3460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbe4b8a25610f345c372f9ad2a1cec4a with result from task 3460 parsl.dataflow.dflow DEBUG: Task 3460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/6e016326-6187-4851-88c6-90d054e4faed_isrForCrosstalkSources_1125916_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/6e016326-6187-4851-88c6-90d054e4faed_isrForCrosstalkSources_1125916_11.stderr parsl.dataflow.dflow INFO: Task 3112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae065cb7361d9cbde2032126c2bd1e39 with result from task 3112 parsl.dataflow.dflow DEBUG: Task 3112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/94194ae5-291b-46c3-a0cc-140438cda44f_isrForCrosstalkSources_1125909_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/94194ae5-291b-46c3-a0cc-140438cda44f_isrForCrosstalkSources_1125909_54.stderr parsl.dataflow.dflow INFO: Task 3421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16e3978d25a5248bc9f13f134e8f826c with result from task 3421 parsl.dataflow.dflow DEBUG: Task 3421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/91b864ba-c347-45f4-bca0-39912788ed7f_isrForCrosstalkSources_1125917_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/91b864ba-c347-45f4-bca0-39912788ed7f_isrForCrosstalkSources_1125917_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:37 proc_lsst.multi:146 [INFO] found job 16896551 in provider checkpoint proc_lsst.multi INFO: found job 16896551 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 526bfe49d77a5cefcfcc482d0e08936f with result from task 3345 parsl.dataflow.dflow DEBUG: Task 3345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/8bd804d3-bf0b-40a1-ab1d-3f34cabe9763_isrForCrosstalkSources_1125920_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/8bd804d3-bf0b-40a1-ab1d-3f34cabe9763_isrForCrosstalkSources_1125920_59.stderr parsl.dataflow.dflow INFO: Task 3274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ca3bf7618b9c7775b1563b049e331d5 with result from task 3274 parsl.dataflow.dflow DEBUG: Task 3274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2584c6cb-2ad7-4825-b9a8-ca12ef573ab1_isrForCrosstalkSources_1125933_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/2584c6cb-2ad7-4825-b9a8-ca12ef573ab1_isrForCrosstalkSources_1125933_44.stderr parsl.dataflow.dflow INFO: Task 3284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 761bb33d6dae6d71322240c054b663e2 with result from task 3284 parsl.dataflow.dflow DEBUG: Task 3284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/399350b1-e543-45e4-834b-68ee9d710cd9_isrForCrosstalkSources_1125924_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/399350b1-e543-45e4-834b-68ee9d710cd9_isrForCrosstalkSources_1125924_59.stderr parsl.dataflow.dflow INFO: Task 3333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d0334b5c26d305e043f1ed05bd9befc with result from task 3333 parsl.dataflow.dflow DEBUG: Task 3333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/ef0ed036-dfd4-4a94-ae8b-aa906d056796_isrForCrosstalkSources_1125912_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/ef0ed036-dfd4-4a94-ae8b-aa906d056796_isrForCrosstalkSources_1125912_52.stderr parsl.dataflow.dflow INFO: Task 3496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c221981532bd1893a1beff17151b334a with result from task 3496 parsl.dataflow.dflow DEBUG: Task 3496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/a7385471-0c9d-4ea3-8f72-fb65eed4489b_isrForCrosstalkSources_1125889_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/a7385471-0c9d-4ea3-8f72-fb65eed4489b_isrForCrosstalkSources_1125889_33.stderr parsl.dataflow.dflow INFO: Task 3484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c0838d3d99b46f106058a25fa1f9a7e with result from task 3484 parsl.dataflow.dflow DEBUG: Task 3484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/81f107f4-40ec-4ac4-89bc-ac1b1b622784_isrForCrosstalkSources_1125916_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/81f107f4-40ec-4ac4-89bc-ac1b1b622784_isrForCrosstalkSources_1125916_22.stderr parsl.dataflow.dflow INFO: Task 3502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0d9356d071bb2dabf11c47ad42deccc with result from task 3502 parsl.dataflow.dflow DEBUG: Task 3502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/57b47812-143e-4c7a-94a6-c2771b459a08_isrForCrosstalkSources_1125896_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/57b47812-143e-4c7a-94a6-c2771b459a08_isrForCrosstalkSources_1125896_14.stderr parsl.dataflow.dflow INFO: Task 3258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0340b7ce8edd4e45d43e1454040b25b7 with result from task 3258 parsl.dataflow.dflow DEBUG: Task 3258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/61eae751-0125-4098-8e7b-e657177c76ec_isrForCrosstalkSources_1125895_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/61eae751-0125-4098-8e7b-e657177c76ec_isrForCrosstalkSources_1125895_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:37 proc_lsst.multi:146 [INFO] found job 16896552 in provider checkpoint proc_lsst.multi INFO: found job 16896552 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c623cfc793280f87e49491403453f57 with result from task 3441 parsl.dataflow.dflow DEBUG: Task 3441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/64d4730c-fb34-4d1c-944e-f0accda31472_isrForCrosstalkSources_1125911_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/64d4730c-fb34-4d1c-944e-f0accda31472_isrForCrosstalkSources_1125911_31.stderr parsl.dataflow.dflow INFO: Task 3382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbd532f80067fef961306cdc41d1af35 with result from task 3382 parsl.dataflow.dflow DEBUG: Task 3382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/87e5b210-b4eb-4b70-b1ce-848b8186a14c_isrForCrosstalkSources_1125918_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/87e5b210-b4eb-4b70-b1ce-848b8186a14c_isrForCrosstalkSources_1125918_60.stderr parsl.dataflow.dflow INFO: Task 3437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 439f3332e014db9eace265b48b652274 with result from task 3437 parsl.dataflow.dflow DEBUG: Task 3437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/2114426e-808d-4f40-b78d-55a866ae4355_isrForCrosstalkSources_1125917_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/2114426e-808d-4f40-b78d-55a866ae4355_isrForCrosstalkSources_1125917_36.stderr parsl.dataflow.dflow INFO: Task 3470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c254fab3803d491a160a161fb6fcf898 with result from task 3470 parsl.dataflow.dflow DEBUG: Task 3470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/6b417d1a-0d6e-4edd-9e52-a86b2dade69c_isrForCrosstalkSources_1125945_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/6b417d1a-0d6e-4edd-9e52-a86b2dade69c_isrForCrosstalkSources_1125945_34.stderr parsl.dataflow.dflow INFO: Task 3472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df288e927b827ff43857d086a0bf37a6 with result from task 3472 parsl.dataflow.dflow DEBUG: Task 3472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/4863f479-61b1-4a36-9d25-422c5366ebdd_isrForCrosstalkSources_1125946_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/4863f479-61b1-4a36-9d25-422c5366ebdd_isrForCrosstalkSources_1125946_60.stderr parsl.dataflow.dflow INFO: Task 3475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a95ef7fd8fd86fb566f7641d2529f8d with result from task 3475 parsl.dataflow.dflow DEBUG: Task 3475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/31984ff4-7073-4254-8c5a-f92273e9aafe_isrForCrosstalkSources_1125932_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/31984ff4-7073-4254-8c5a-f92273e9aafe_isrForCrosstalkSources_1125932_15.stderr parsl.dataflow.dflow INFO: Task 3406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6737e25449a3c938e50e3712a8608af2 with result from task 3406 parsl.dataflow.dflow DEBUG: Task 3406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/5c5a363a-d868-444b-be67-bc38b4d89aab_isrForCrosstalkSources_1125906_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/5c5a363a-d868-444b-be67-bc38b4d89aab_isrForCrosstalkSources_1125906_26.stderr parsl.dataflow.dflow INFO: Task 3398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e09059f3b005c841369fc41fade20a47 with result from task 3398 parsl.dataflow.dflow DEBUG: Task 3398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/0726c36f-ae2c-417b-8772-7a6c1bd1d133_isrForCrosstalkSources_1125893_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/0726c36f-ae2c-417b-8772-7a6c1bd1d133_isrForCrosstalkSources_1125893_54.stderr parsl.dataflow.dflow INFO: Task 3467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17486da978ab83fdc9b89943465ff6a8 with result from task 3467 parsl.dataflow.dflow DEBUG: Task 3467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/e89014a9-5190-4c1f-84b1-1e3218a6fc6e_isrForCrosstalkSources_1125911_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/e89014a9-5190-4c1f-84b1-1e3218a6fc6e_isrForCrosstalkSources_1125911_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:38 proc_lsst.multi:146 [INFO] found job 16896553 in provider checkpoint proc_lsst.multi INFO: found job 16896553 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61eb305b2a760b3a96e9d3370ec2a986 with result from task 3486 parsl.dataflow.dflow DEBUG: Task 3486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/eb55feea-fbc6-4b2e-9de8-ee276840041d_isrForCrosstalkSources_1125894_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/eb55feea-fbc6-4b2e-9de8-ee276840041d_isrForCrosstalkSources_1125894_45.stderr parsl.dataflow.dflow INFO: Task 3378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f8c5b841b67af6dc818ab4e063f6b23 with result from task 3378 parsl.dataflow.dflow DEBUG: Task 3378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/5580be57-63ac-44b9-959e-d30f85cef70b_isrForCrosstalkSources_1125940_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/5580be57-63ac-44b9-959e-d30f85cef70b_isrForCrosstalkSources_1125940_53.stderr parsl.dataflow.dflow INFO: Task 3400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9547e8c7cb3f83e66816df5b702f83f4 with result from task 3400 parsl.dataflow.dflow DEBUG: Task 3400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/713ee1ef-5602-481f-ad00-4b7abe577bfd_isrForCrosstalkSources_1125908_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/713ee1ef-5602-481f-ad00-4b7abe577bfd_isrForCrosstalkSources_1125908_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3109 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 5ae026d7d0f523bf7af19b8385fc7b32 with result from task 3109 parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3109 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/c4f0071d-bf1f-4d5a-aee6-b7b2d530cf88_isrForCrosstalkSources_1125942_43.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/c4f0071d-bf1f-4d5a-aee6-b7b2d530cf88_isrForCrosstalkSources_1125942_43.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:38 proc_lsst.multi:146 [INFO] found job 16896554 in provider checkpoint proc_lsst.multi INFO: found job 16896554 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2123f7092ebed643b9034b88126ec2b3 with result from task 3434 parsl.dataflow.dflow DEBUG: Task 3434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/1062ed0f-81dd-4ba2-a045-f738ecf90049_isrForCrosstalkSources_1125905_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/1062ed0f-81dd-4ba2-a045-f738ecf90049_isrForCrosstalkSources_1125905_56.stderr parsl.dataflow.dflow INFO: Task 3455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95957149d59b2ff997d39af96a2e1554 with result from task 3455 parsl.dataflow.dflow DEBUG: Task 3455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/9e1ef4e1-23cb-4596-a798-2d82b60490b1_isrForCrosstalkSources_1125888_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/9e1ef4e1-23cb-4596-a798-2d82b60490b1_isrForCrosstalkSources_1125888_56.stderr parsl.dataflow.dflow INFO: Task 3449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33a4e4aade829af6426c59ba30f7725e with result from task 3449 parsl.dataflow.dflow DEBUG: Task 3449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/4ea113d4-5926-43f1-b31a-44a79d799ddc_isrForCrosstalkSources_1125925_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/4ea113d4-5926-43f1-b31a-44a79d799ddc_isrForCrosstalkSources_1125925_38.stderr parsl.dataflow.dflow INFO: Task 3451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57e67077425a3173f0fc0a620d81523b with result from task 3451 parsl.dataflow.dflow DEBUG: Task 3451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/805adfd4-8de2-4e17-9839-52e046bc8ca4_isrForCrosstalkSources_1125923_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/805adfd4-8de2-4e17-9839-52e046bc8ca4_isrForCrosstalkSources_1125923_43.stderr parsl.dataflow.dflow INFO: Task 3448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adb248b1660e24f9db811dd864222463 with result from task 3448 parsl.dataflow.dflow DEBUG: Task 3448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/0b33caa4-04df-4bed-90c0-dd4de45a5e0d_isrForCrosstalkSources_1125905_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/0b33caa4-04df-4bed-90c0-dd4de45a5e0d_isrForCrosstalkSources_1125905_40.stderr parsl.dataflow.dflow INFO: Task 3457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a83f589dcdf6ebeb282d17a99a327a33 with result from task 3457 parsl.dataflow.dflow DEBUG: Task 3457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/4c04d0ad-25c8-4a2e-87fa-b04a5083ac09_isrForCrosstalkSources_1125924_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/4c04d0ad-25c8-4a2e-87fa-b04a5083ac09_isrForCrosstalkSources_1125924_53.stderr parsl.dataflow.dflow INFO: Task 3442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0607fd2e0cd45f3242e2c89aa38ab318 with result from task 3442 parsl.dataflow.dflow DEBUG: Task 3442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/25f3a7e9-3fb4-4b71-9060-a9066aec21e1_isrForCrosstalkSources_1125939_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/25f3a7e9-3fb4-4b71-9060-a9066aec21e1_isrForCrosstalkSources_1125939_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:38 proc_lsst.multi:146 [INFO] found job 16896555 in provider checkpoint proc_lsst.multi INFO: found job 16896555 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab88ce6cc40a35d12aaa7c0ab105c867 with result from task 3466 parsl.dataflow.dflow DEBUG: Task 3466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/54b4a54e-74a5-4278-a813-59ce0a5149c9_isrForCrosstalkSources_1125905_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/54b4a54e-74a5-4278-a813-59ce0a5149c9_isrForCrosstalkSources_1125905_54.stderr parsl.dataflow.dflow INFO: Task 3473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2e135911e063c763ea0be6e577a8daa with result from task 3473 parsl.dataflow.dflow DEBUG: Task 3473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/31ad08f6-5047-499c-bb33-b014a038b0b6_isrForCrosstalkSources_1125896_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/31ad08f6-5047-499c-bb33-b014a038b0b6_isrForCrosstalkSources_1125896_17.stderr parsl.dataflow.dflow INFO: Task 3113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45441fa33ab0d755c4d1c32ed49abf43 with result from task 3113 parsl.dataflow.dflow DEBUG: Task 3113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/7385d986-7223-495a-8474-577680892f9c_isrForCrosstalkSources_1125923_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/7385d986-7223-495a-8474-577680892f9c_isrForCrosstalkSources_1125923_42.stderr parsl.dataflow.dflow INFO: Task 3139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b27809ec18be3773e923f575aaab3391 with result from task 3139 parsl.dataflow.dflow DEBUG: Task 3139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/a9564c54-8587-4b52-937f-04c12414e016_isrForCrosstalkSources_1125912_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/a9564c54-8587-4b52-937f-04c12414e016_isrForCrosstalkSources_1125912_38.stderr parsl.dataflow.dflow INFO: Task 3447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc71062b824de8a5c401c4771dc48cdb with result from task 3447 parsl.dataflow.dflow DEBUG: Task 3447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/6cfa29eb-e7bb-475d-9732-7de03e382274_isrForCrosstalkSources_1125947_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/6cfa29eb-e7bb-475d-9732-7de03e382274_isrForCrosstalkSources_1125947_10.stderr parsl.dataflow.dflow INFO: Task 3155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b47ba3cd31a7675e3893d6f037cd83a with result from task 3155 parsl.dataflow.dflow DEBUG: Task 3155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/eca60bcf-59e7-421b-bb64-ddf510d849d7_isrForCrosstalkSources_1125941_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/eca60bcf-59e7-421b-bb64-ddf510d849d7_isrForCrosstalkSources_1125941_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:38 proc_lsst.multi:146 [INFO] found job 16896556 in provider checkpoint proc_lsst.multi INFO: found job 16896556 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef2059854ab44a63d0b1ac7e7e8a3d56 with result from task 3150 parsl.dataflow.dflow DEBUG: Task 3150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/7e4f636f-6caa-4fcb-bef5-4bd082457c6b_isrForCrosstalkSources_1125932_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/7e4f636f-6caa-4fcb-bef5-4bd082457c6b_isrForCrosstalkSources_1125932_41.stderr parsl.dataflow.dflow INFO: Task 3145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 753406a33bc8315e0f16039e3f602e14 with result from task 3145 parsl.dataflow.dflow DEBUG: Task 3145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/9e71dd50-d9a4-4dc9-80cc-ecf822acac69_isrForCrosstalkSources_1125944_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/9e71dd50-d9a4-4dc9-80cc-ecf822acac69_isrForCrosstalkSources_1125944_58.stderr parsl.dataflow.dflow INFO: Task 3468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f26243ed7ab32338ffdb7de392b513e with result from task 3468 parsl.dataflow.dflow DEBUG: Task 3468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/cab403f1-d0c0-4242-b070-1e5071dd2b5d_isrForCrosstalkSources_1125917_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/cab403f1-d0c0-4242-b070-1e5071dd2b5d_isrForCrosstalkSources_1125917_35.stderr parsl.dataflow.dflow INFO: Task 3458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c307fe6e7bee347a793d080feef31bc1 with result from task 3458 parsl.dataflow.dflow DEBUG: Task 3458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/13eceab6-e4bf-4dca-b911-9794c54d8f37_isrForCrosstalkSources_1125895_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/13eceab6-e4bf-4dca-b911-9794c54d8f37_isrForCrosstalkSources_1125895_47.stderr parsl.dataflow.dflow INFO: Task 3390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 300cbaf00c8054ceee2abf36153aa420 with result from task 3390 parsl.dataflow.dflow DEBUG: Task 3390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/1f36cde0-3f0e-481d-9129-5976dd5d9c0e_isrForCrosstalkSources_1125918_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/1f36cde0-3f0e-481d-9129-5976dd5d9c0e_isrForCrosstalkSources_1125918_59.stderr parsl.dataflow.dflow INFO: Task 3127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4364d602be38f92617a20f952becd275 with result from task 3127 parsl.dataflow.dflow DEBUG: Task 3127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/0ab5f648-0de8-46be-9076-052c68e804bd_isrForCrosstalkSources_1125900_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/0ab5f648-0de8-46be-9076-052c68e804bd_isrForCrosstalkSources_1125900_35.stderr parsl.dataflow.dflow INFO: Task 3456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 175867012f2db6613045bbda645db879 with result from task 3456 parsl.dataflow.dflow DEBUG: Task 3456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/b341795c-594d-4213-bdfe-8f92dc040cc4_isrForCrosstalkSources_1125936_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/b341795c-594d-4213-bdfe-8f92dc040cc4_isrForCrosstalkSources_1125936_62.stderr parsl.dataflow.dflow INFO: Task 3149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44a2712a8dc02626d8cc5553fb68d93f with result from task 3149 parsl.dataflow.dflow DEBUG: Task 3149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/6c5571ea-c485-400d-9aac-bf9bb3d1da5e_isrForCrosstalkSources_1125923_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/6c5571ea-c485-400d-9aac-bf9bb3d1da5e_isrForCrosstalkSources_1125923_44.stderr parsl.dataflow.dflow INFO: Task 3481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab0a6b130aeef8f2475476863efdb700 with result from task 3481 parsl.dataflow.dflow DEBUG: Task 3481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/6762db47-5f10-455e-8e7e-782aba2df2aa_isrForCrosstalkSources_1125899_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/6762db47-5f10-455e-8e7e-782aba2df2aa_isrForCrosstalkSources_1125899_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:38 proc_lsst.multi:146 [INFO] found job 16896559 in provider checkpoint proc_lsst.multi INFO: found job 16896559 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b3b9c1065deb144760bfe831b201a9b with result from task 3450 parsl.dataflow.dflow DEBUG: Task 3450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/676d6433-dcf0-4db2-8690-74ccacd64fd2_isrForCrosstalkSources_1125900_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/676d6433-dcf0-4db2-8690-74ccacd64fd2_isrForCrosstalkSources_1125900_57.stderr parsl.dataflow.dflow INFO: Task 3154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb8f5239f1021f89285e9ba3f3ebaa25 with result from task 3154 parsl.dataflow.dflow DEBUG: Task 3154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/de9f5173-5cd9-480a-9b80-b50dd2113a64_isrForCrosstalkSources_1125888_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/de9f5173-5cd9-480a-9b80-b50dd2113a64_isrForCrosstalkSources_1125888_60.stderr parsl.dataflow.dflow INFO: Task 3510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7680c3a19962425713efb3ef40ad62da with result from task 3510 parsl.dataflow.dflow DEBUG: Task 3510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/ba284490-deb8-4b09-9482-ac1c4592c583_isrForCrosstalkSources_1125892_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/ba284490-deb8-4b09-9482-ac1c4592c583_isrForCrosstalkSources_1125892_34.stderr parsl.dataflow.dflow INFO: Task 3487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4197ecf28399fbe38aaee2fa847a2ece with result from task 3487 parsl.dataflow.dflow DEBUG: Task 3487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/97c52b5e-7e75-4b33-b666-13c6bca4640e_isrForCrosstalkSources_1125913_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/97c52b5e-7e75-4b33-b666-13c6bca4640e_isrForCrosstalkSources_1125913_45.stderr parsl.dataflow.dflow INFO: Task 3445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1623747594505e040f65453b386dbf4d with result from task 3445 parsl.dataflow.dflow DEBUG: Task 3445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/f977f2cc-9cdb-488d-beba-95d95e8cb8cc_isrForCrosstalkSources_1125943_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/f977f2cc-9cdb-488d-beba-95d95e8cb8cc_isrForCrosstalkSources_1125943_4.stderr parsl.dataflow.dflow INFO: Task 3514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ba9b418bc627f63340be63a6087f7db with result from task 3514 parsl.dataflow.dflow DEBUG: Task 3514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/d9deaaee-a865-48fa-9c8a-eba7fbf655fd_isrForCrosstalkSources_1125933_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/d9deaaee-a865-48fa-9c8a-eba7fbf655fd_isrForCrosstalkSources_1125933_32.stderr parsl.dataflow.dflow INFO: Task 3503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fe0ccdd54976dd2ae6ac7ab3f6d0b84 with result from task 3503 parsl.dataflow.dflow DEBUG: Task 3503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/6415c376-1d3d-462a-9529-6e43e520ac88_isrForCrosstalkSources_1125941_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/6415c376-1d3d-462a-9529-6e43e520ac88_isrForCrosstalkSources_1125941_11.stderr parsl.dataflow.dflow INFO: Task 3518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8253fbf5081765a96a6a3ae28c39a89d with result from task 3518 parsl.dataflow.dflow DEBUG: Task 3518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/ffc56801-20de-4f0b-b5f5-dcc07676a56f_isrForCrosstalkSources_1125928_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/ffc56801-20de-4f0b-b5f5-dcc07676a56f_isrForCrosstalkSources_1125928_35.stderr parsl.dataflow.dflow INFO: Task 3504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1259f0ecb557b206fb50cfc6e80c3a7d with result from task 3504 parsl.dataflow.dflow DEBUG: Task 3504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/1b963527-c491-4fd8-8e8b-8b7bcbfb3f2b_isrForCrosstalkSources_1125920_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/1b963527-c491-4fd8-8e8b-8b7bcbfb3f2b_isrForCrosstalkSources_1125920_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:39 proc_lsst.multi:146 [INFO] found job 16896561 in provider checkpoint proc_lsst.multi INFO: found job 16896561 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f43afb67e04aac26a2b93527de2a3255 with result from task 3530 parsl.dataflow.dflow DEBUG: Task 3530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/c1509af9-97cd-4ca2-8a94-040d8a1218e4_isrForCrosstalkSources_1125918_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/c1509af9-97cd-4ca2-8a94-040d8a1218e4_isrForCrosstalkSources_1125918_42.stderr parsl.dataflow.dflow INFO: Task 3482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57887c6aed2828d0baf2d8f5e2e3b319 with result from task 3482 parsl.dataflow.dflow DEBUG: Task 3482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/83e1bf6c-683b-41e4-b873-b675edb3ee76_isrForCrosstalkSources_1125891_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/83e1bf6c-683b-41e4-b873-b675edb3ee76_isrForCrosstalkSources_1125891_20.stderr parsl.dataflow.dflow INFO: Task 3418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee5f3515bd0619f818a08eb5bfd03a41 with result from task 3418 parsl.dataflow.dflow DEBUG: Task 3418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/44d29fcf-46e9-43b4-9c77-d9c29201bf88_isrForCrosstalkSources_1125924_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/44d29fcf-46e9-43b4-9c77-d9c29201bf88_isrForCrosstalkSources_1125924_39.stderr parsl.dataflow.dflow INFO: Task 3423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd7c80e9505f670f468bb7d230fc0995 with result from task 3423 parsl.dataflow.dflow DEBUG: Task 3423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/af44c822-a9e4-4038-8f17-f2a0ccef9b5c_isrForCrosstalkSources_1125924_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/af44c822-a9e4-4038-8f17-f2a0ccef9b5c_isrForCrosstalkSources_1125924_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:40 proc_lsst.multi:146 [INFO] found job 16896562 in provider checkpoint proc_lsst.multi INFO: found job 16896562 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b24cb269ecd87d9cae192e978b768fdf with result from task 3477 parsl.dataflow.dflow DEBUG: Task 3477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/5060b7f1-411b-443c-b3eb-1319893a7bbd_isrForCrosstalkSources_1125892_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/5060b7f1-411b-443c-b3eb-1319893a7bbd_isrForCrosstalkSources_1125892_46.stderr parsl.dataflow.dflow INFO: Task 3523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba3d04c3fd3a6e246b96cff0ef770b16 with result from task 3523 parsl.dataflow.dflow DEBUG: Task 3523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/82f2dea5-fb52-46db-90c4-00aa3afce70f_isrForCrosstalkSources_1125918_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/82f2dea5-fb52-46db-90c4-00aa3afce70f_isrForCrosstalkSources_1125918_1.stderr parsl.dataflow.dflow INFO: Task 3408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8378aa07776820ff49b2ee4c6f84e306 with result from task 3408 parsl.dataflow.dflow DEBUG: Task 3408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/1618d355-5c09-4c5b-bff0-f25cb222fa41_isrForCrosstalkSources_1125901_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/1618d355-5c09-4c5b-bff0-f25cb222fa41_isrForCrosstalkSources_1125901_15.stderr parsl.dataflow.dflow INFO: Task 3459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 645c2bc48f39bda1ece4685a58e97202 with result from task 3459 parsl.dataflow.dflow DEBUG: Task 3459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/38453ed6-fe65-4585-a6c0-442860505155_isrForCrosstalkSources_1125890_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/38453ed6-fe65-4585-a6c0-442860505155_isrForCrosstalkSources_1125890_23.stderr parsl.dataflow.dflow INFO: Task 3516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2780d470e31082f4acbb53cdc049eac with result from task 3516 parsl.dataflow.dflow DEBUG: Task 3516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/efe1768a-b73e-4d07-bee9-907b5e3f358e_isrForCrosstalkSources_1125901_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/efe1768a-b73e-4d07-bee9-907b5e3f358e_isrForCrosstalkSources_1125901_18.stderr parsl.dataflow.dflow INFO: Task 3562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4eac50d2f230f90dd4171308164d5ccf with result from task 3562 parsl.dataflow.dflow DEBUG: Task 3562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9c0ead5b-6e52-4cf6-a3a3-6169bb591688_isrForCrosstalkSources_1125934_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/9c0ead5b-6e52-4cf6-a3a3-6169bb591688_isrForCrosstalkSources_1125934_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:40 proc_lsst.multi:146 [INFO] found job 16896563 in provider checkpoint proc_lsst.multi INFO: found job 16896563 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6e0f4879ec72b73b34f255949be0321 with result from task 3541 parsl.dataflow.dflow DEBUG: Task 3541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/e8118866-cd33-4945-a6e9-6515b95333d4_isrForCrosstalkSources_1125923_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/e8118866-cd33-4945-a6e9-6515b95333d4_isrForCrosstalkSources_1125923_3.stderr parsl.dataflow.dflow INFO: Task 3539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry feb2817069151a8cacd89893aeb0a152 with result from task 3539 parsl.dataflow.dflow DEBUG: Task 3539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/601509f2-dbcd-4aec-9941-29573b96e6bb_isrForCrosstalkSources_1125928_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/601509f2-dbcd-4aec-9941-29573b96e6bb_isrForCrosstalkSources_1125928_4.stderr parsl.dataflow.dflow INFO: Task 3461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f15d78ec40863a46db7770fc694824c6 with result from task 3461 parsl.dataflow.dflow DEBUG: Task 3461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/f5a4e545-2e0b-436b-b6ed-f86a621041fa_isrForCrosstalkSources_1125928_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/f5a4e545-2e0b-436b-b6ed-f86a621041fa_isrForCrosstalkSources_1125928_29.stderr parsl.dataflow.dflow INFO: Task 3513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6a73fbfd40168aa72071d1df3ab3968 with result from task 3513 parsl.dataflow.dflow DEBUG: Task 3513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/ec682543-fbf7-4e42-aa87-b06edb662bb7_isrForCrosstalkSources_1125938_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/ec682543-fbf7-4e42-aa87-b06edb662bb7_isrForCrosstalkSources_1125938_10.stderr parsl.dataflow.dflow INFO: Task 3508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60e30a28bc4943d9f626e78aaadb222b with result from task 3508 parsl.dataflow.dflow DEBUG: Task 3508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/f19df8a7-ae56-4433-b895-3f4931f51c21_isrForCrosstalkSources_1125908_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/f19df8a7-ae56-4433-b895-3f4931f51c21_isrForCrosstalkSources_1125908_12.stderr parsl.dataflow.dflow INFO: Task 3465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e353a8ad991f0e38eb4bd6210b0a20c4 with result from task 3465 parsl.dataflow.dflow DEBUG: Task 3465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/6b8b957f-dc8a-47b4-a651-4cac02629fe0_isrForCrosstalkSources_1125906_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/6b8b957f-dc8a-47b4-a651-4cac02629fe0_isrForCrosstalkSources_1125906_27.stderr parsl.dataflow.dflow INFO: Task 3474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2f61800e97448bc26a7b21e051b21fc with result from task 3474 parsl.dataflow.dflow DEBUG: Task 3474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/090c22b1-57b6-4766-ac88-2b2c8a69e276_isrForCrosstalkSources_1125909_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/090c22b1-57b6-4766-ac88-2b2c8a69e276_isrForCrosstalkSources_1125909_19.stderr parsl.dataflow.dflow INFO: Task 3537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da1fdbe9c300dd83a49a23e0a513f785 with result from task 3537 parsl.dataflow.dflow DEBUG: Task 3537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/f815da8a-82f2-4f9e-beee-3c921b30c0ff_isrForCrosstalkSources_1125942_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/f815da8a-82f2-4f9e-beee-3c921b30c0ff_isrForCrosstalkSources_1125942_38.stderr parsl.dataflow.dflow INFO: Task 3387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69f47d2bf1cf4160ee1ca5cc0318cd72 with result from task 3387 parsl.dataflow.dflow DEBUG: Task 3387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/a2f482b0-8c8a-4ac5-b6b9-2a6df27a86a8_isrForCrosstalkSources_1125908_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/a2f482b0-8c8a-4ac5-b6b9-2a6df27a86a8_isrForCrosstalkSources_1125908_5.stderr parsl.dataflow.dflow INFO: Task 3420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b39863695b8bca8cd548957a1bafd8de with result from task 3420 parsl.dataflow.dflow DEBUG: Task 3420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/2472105b-3f16-47b7-b051-4d3e3f7c1f81_isrForCrosstalkSources_1125904_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/2472105b-3f16-47b7-b051-4d3e3f7c1f81_isrForCrosstalkSources_1125904_12.stderr parsl.dataflow.dflow INFO: Task 3417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06f82f3da1b50b3cb756c27e34b84218 with result from task 3417 parsl.dataflow.dflow DEBUG: Task 3417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/10290e67-89c3-4394-bf33-328ddd62a096_isrForCrosstalkSources_1125921_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/10290e67-89c3-4394-bf33-328ddd62a096_isrForCrosstalkSources_1125921_49.stderr parsl.dataflow.dflow INFO: Task 3544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60acb94872ba42c5c1a7accaa7aba98c with result from task 3544 parsl.dataflow.dflow DEBUG: Task 3544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d82445eb-dadc-4b0d-98c0-37012293459b_isrForCrosstalkSources_1125915_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d82445eb-dadc-4b0d-98c0-37012293459b_isrForCrosstalkSources_1125915_11.stderr parsl.dataflow.dflow INFO: Task 3498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10c53f8b7f61806f00f2eda40f3aea1f with result from task 3498 parsl.dataflow.dflow DEBUG: Task 3498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/c66c43e1-0ec7-402e-a096-41d53264dc3c_isrForCrosstalkSources_1125900_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/c66c43e1-0ec7-402e-a096-41d53264dc3c_isrForCrosstalkSources_1125900_16.stderr parsl.dataflow.dflow INFO: Task 3499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51cfc3ff0184b0b733b044dcdd9bfa93 with result from task 3499 parsl.dataflow.dflow DEBUG: Task 3499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/25b7e408-e1c3-4b20-a493-573c97cac4d6_isrForCrosstalkSources_1125921_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/25b7e408-e1c3-4b20-a493-573c97cac4d6_isrForCrosstalkSources_1125921_46.stderr parsl.dataflow.dflow INFO: Task 3553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab8ecd725b37ca20ca5fff7e670faa10 with result from task 3553 parsl.dataflow.dflow DEBUG: Task 3553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f621b493-ead9-43af-9cd4-dc681002a098_isrForCrosstalkSources_1125910_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125910/f621b493-ead9-43af-9cd4-dc681002a098_isrForCrosstalkSources_1125910_11.stderr parsl.dataflow.dflow INFO: Task 3485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efc7a083454ab0422a14c78f442973d2 with result from task 3485 parsl.dataflow.dflow DEBUG: Task 3485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/a4cfb9f2-f704-435e-b9c8-6aece0a642ad_isrForCrosstalkSources_1125888_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/a4cfb9f2-f704-435e-b9c8-6aece0a642ad_isrForCrosstalkSources_1125888_31.stderr parsl.dataflow.dflow INFO: Task 3391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 912c449e414442b1ed73364996ba08f6 with result from task 3391 parsl.dataflow.dflow DEBUG: Task 3391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/1715b1ce-1365-4645-964d-8ab8db660ad4_isrForCrosstalkSources_1125921_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/1715b1ce-1365-4645-964d-8ab8db660ad4_isrForCrosstalkSources_1125921_48.stderr parsl.dataflow.dflow INFO: Task 3443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6b7f9e6f570db53cd60dc53c24d7125 with result from task 3443 parsl.dataflow.dflow DEBUG: Task 3443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/2be33fbe-103b-430a-832e-582181f6e518_isrForCrosstalkSources_1125944_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/2be33fbe-103b-430a-832e-582181f6e518_isrForCrosstalkSources_1125944_43.stderr parsl.dataflow.dflow INFO: Task 3401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65c3db299ef66cf4d8e7111d8f76b536 with result from task 3401 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:41 proc_lsst.multi:146 [INFO] found job 16896564 in provider checkpoint proc_lsst.multi INFO: found job 16896564 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 3401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/5c1063c7-e9a7-4d61-8011-5e693e77f1f4_isrForCrosstalkSources_1125936_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/5c1063c7-e9a7-4d61-8011-5e693e77f1f4_isrForCrosstalkSources_1125936_7.stderr parsl.dataflow.dflow INFO: Task 3395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8acb4a6c4315d41811c9b5aa070e2a82 with result from task 3395 parsl.dataflow.dflow DEBUG: Task 3395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/76e9e203-ea93-4a62-a42a-99cf78d8c9d9_isrForCrosstalkSources_1125927_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/76e9e203-ea93-4a62-a42a-99cf78d8c9d9_isrForCrosstalkSources_1125927_15.stderr parsl.dataflow.dflow INFO: Task 3363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ae1f31b035356b2e4aa9963b5de508f with result from task 3363 parsl.dataflow.dflow DEBUG: Task 3363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/8cb30c4b-b49f-431c-9602-c1726d49c3c6_isrForCrosstalkSources_1125937_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/8cb30c4b-b49f-431c-9602-c1726d49c3c6_isrForCrosstalkSources_1125937_6.stderr parsl.dataflow.dflow INFO: Task 3524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fa4e54fd39db7c7611b9199a782e02f with result from task 3524 parsl.dataflow.dflow DEBUG: Task 3524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/c4fff655-2d6f-4dbf-a862-e9574660cfe8_isrForCrosstalkSources_1125888_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/c4fff655-2d6f-4dbf-a862-e9574660cfe8_isrForCrosstalkSources_1125888_4.stderr parsl.dataflow.dflow INFO: Task 3526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88fa018557fa039a8e26754e4d2c45f0 with result from task 3526 parsl.dataflow.dflow DEBUG: Task 3526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/9e998866-691e-42e9-8b69-5c634e86c6a4_isrForCrosstalkSources_1125922_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/9e998866-691e-42e9-8b69-5c634e86c6a4_isrForCrosstalkSources_1125922_12.stderr parsl.dataflow.dflow INFO: Task 3462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76ead707f87f3364bb9b93e7817345fa with result from task 3462 parsl.dataflow.dflow DEBUG: Task 3462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/a0f18f4b-aaab-4796-851d-b1336300e0ba_isrForCrosstalkSources_1125916_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/a0f18f4b-aaab-4796-851d-b1336300e0ba_isrForCrosstalkSources_1125916_37.stderr parsl.dataflow.dflow INFO: Task 3569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b9dcefe6195e0f4bf856aa3a0dd559b with result from task 3569 parsl.dataflow.dflow DEBUG: Task 3569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/4137e972-e04a-4893-9294-340ce49579d3_isrForCrosstalkSources_1125916_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/4137e972-e04a-4893-9294-340ce49579d3_isrForCrosstalkSources_1125916_56.stderr parsl.dataflow.dflow INFO: Task 3389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee848bc1774fd15d94133d90f910e250 with result from task 3389 parsl.dataflow.dflow DEBUG: Task 3389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c971b011-a26c-45dd-ba44-945a85f38de2_isrForCrosstalkSources_1125916_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/c971b011-a26c-45dd-ba44-945a85f38de2_isrForCrosstalkSources_1125916_28.stderr parsl.dataflow.dflow INFO: Task 3404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09cfda5359081358b25117016774c86c with result from task 3404 parsl.dataflow.dflow DEBUG: Task 3404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/12eaf46e-7779-4243-ba3c-a60260c62afa_isrForCrosstalkSources_1125932_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/12eaf46e-7779-4243-ba3c-a60260c62afa_isrForCrosstalkSources_1125932_8.stderr parsl.dataflow.dflow INFO: Task 3433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9307736bc168f3f5d074c2bea10a2d96 with result from task 3433 parsl.dataflow.dflow DEBUG: Task 3433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/43d5f9ce-3fff-44dc-aac5-afbc1ec39669_isrForCrosstalkSources_1125912_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/43d5f9ce-3fff-44dc-aac5-afbc1ec39669_isrForCrosstalkSources_1125912_40.stderr parsl.dataflow.dflow INFO: Task 3428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1e5f9d9c14437e31c8bad4f663a9d06 with result from task 3428 parsl.dataflow.dflow DEBUG: Task 3428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/5f09e175-ed7c-4fee-9c97-170bac8bddcc_isrForCrosstalkSources_1125942_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/5f09e175-ed7c-4fee-9c97-170bac8bddcc_isrForCrosstalkSources_1125942_8.stderr parsl.dataflow.dflow INFO: Task 3500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af0c2ea29e75a8e673cb519ea9a30c16 with result from task 3500 parsl.dataflow.dflow DEBUG: Task 3500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/bcdbeaad-2745-4deb-8509-6162125f6e58_isrForCrosstalkSources_1125908_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125908/bcdbeaad-2745-4deb-8509-6162125f6e58_isrForCrosstalkSources_1125908_36.stderr parsl.dataflow.dflow INFO: Task 3419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99784a3231a7f74fdf26aedf394a9117 with result from task 3419 parsl.dataflow.dflow DEBUG: Task 3419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/bc4773a0-a071-46a3-abff-862f8e1a67f9_isrForCrosstalkSources_1125947_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/bc4773a0-a071-46a3-abff-862f8e1a67f9_isrForCrosstalkSources_1125947_30.stderr parsl.dataflow.dflow INFO: Task 3453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56b28a1c12950b0d8e090b6b3f34f094 with result from task 3453 parsl.dataflow.dflow DEBUG: Task 3453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/028fb13a-9e51-4d92-b9d5-f3d87f23f0ef_isrForCrosstalkSources_1125902_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/028fb13a-9e51-4d92-b9d5-f3d87f23f0ef_isrForCrosstalkSources_1125902_41.stderr parsl.dataflow.dflow INFO: Task 3431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb513c919159743fac8c81405469901c with result from task 3431 parsl.dataflow.dflow DEBUG: Task 3431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/8452bea1-b019-4935-b8e2-1a0e3f16738f_isrForCrosstalkSources_1125939_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/8452bea1-b019-4935-b8e2-1a0e3f16738f_isrForCrosstalkSources_1125939_39.stderr parsl.dataflow.dflow INFO: Task 3501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 382d62d9e42fc89e028c299e41332849 with result from task 3501 parsl.dataflow.dflow DEBUG: Task 3501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/473f7f46-6592-48ab-a851-6e21c306f878_isrForCrosstalkSources_1125916_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/473f7f46-6592-48ab-a851-6e21c306f878_isrForCrosstalkSources_1125916_20.stderr parsl.dataflow.dflow INFO: Task 3547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 361fea3f40dba9f925765f894759f7e7 with result from task 3547 parsl.dataflow.dflow DEBUG: Task 3547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/303a5ef6-ea65-4b04-b504-83b40b866613_isrForCrosstalkSources_1125896_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/303a5ef6-ea65-4b04-b504-83b40b866613_isrForCrosstalkSources_1125896_13.stderr parsl.dataflow.dflow INFO: Task 3489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d413c36b64b9cc47cdd8ba02875a4fba with result from task 3489 parsl.dataflow.dflow DEBUG: Task 3489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/95bd77a8-1d86-4598-b01f-e51b459ef0ec_isrForCrosstalkSources_1125947_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/95bd77a8-1d86-4598-b01f-e51b459ef0ec_isrForCrosstalkSources_1125947_1.stderr parsl.dataflow.dflow INFO: Task 3488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d040032ebb59182615f8e7e8086f5ca with result from task 3488 parsl.dataflow.dflow DEBUG: Task 3488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8f023870-690f-405f-a801-e12d4ae35ab4_isrForCrosstalkSources_1125944_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/8f023870-690f-405f-a801-e12d4ae35ab4_isrForCrosstalkSources_1125944_46.stderr parsl.dataflow.dflow INFO: Task 3509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a3e4f293772675f1d5853a3c9bbcf1f with result from task 3509 parsl.dataflow.dflow DEBUG: Task 3509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/c457e914-6f27-40df-8c7c-7a5da6d736cf_isrForCrosstalkSources_1125943_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/c457e914-6f27-40df-8c7c-7a5da6d736cf_isrForCrosstalkSources_1125943_18.stderr parsl.dataflow.dflow INFO: Task 3572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2661289af354900a891ebc615dc55b2 with result from task 3572 parsl.dataflow.dflow DEBUG: Task 3572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/e05445d0-c7fa-4423-adcc-8690a4b41d2e_isrForCrosstalkSources_1125928_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/e05445d0-c7fa-4423-adcc-8690a4b41d2e_isrForCrosstalkSources_1125928_22.stderr parsl.dataflow.dflow INFO: Task 3497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b6c31fd3770c73cac1bb89fce8d4351 with result from task 3497 parsl.dataflow.dflow DEBUG: Task 3497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/838037ac-388d-4853-987d-dd8c04e43b25_isrForCrosstalkSources_1125936_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/838037ac-388d-4853-987d-dd8c04e43b25_isrForCrosstalkSources_1125936_47.stderr parsl.dataflow.dflow INFO: Task 3597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4194eb149b84a5a09a42bc28836810d9 with result from task 3597 parsl.dataflow.dflow DEBUG: Task 3597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f8743036-924f-4be1-8b7e-2d7daafeff7a_isrForCrosstalkSources_1125945_40.stdout parsl.dataflow.dflow INFO: Standard error for task 3597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/f8743036-924f-4be1-8b7e-2d7daafeff7a_isrForCrosstalkSources_1125945_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:41 proc_lsst.multi:146 [INFO] found job 16896565 in provider checkpoint proc_lsst.multi INFO: found job 16896565 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2f4f4675277649b4bf35c8d3e67522f with result from task 3517 parsl.dataflow.dflow DEBUG: Task 3517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/5c74197b-c6d0-43ff-abef-20bd7779bdf7_isrForCrosstalkSources_1125895_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/5c74197b-c6d0-43ff-abef-20bd7779bdf7_isrForCrosstalkSources_1125895_49.stderr parsl.dataflow.dflow INFO: Task 3507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d52e9b75a954577ac88b9a6611be7556 with result from task 3507 parsl.dataflow.dflow DEBUG: Task 3507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/66d34d2e-9d9e-49d4-9582-166998eb0f3b_isrForCrosstalkSources_1125933_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/66d34d2e-9d9e-49d4-9582-166998eb0f3b_isrForCrosstalkSources_1125933_62.stderr parsl.dataflow.dflow INFO: Task 3520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bbe908812979eaf829d84ef925d1bc9 with result from task 3520 parsl.dataflow.dflow DEBUG: Task 3520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/ef353909-3943-4451-b70a-019e4f9b7b45_isrForCrosstalkSources_1125899_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/ef353909-3943-4451-b70a-019e4f9b7b45_isrForCrosstalkSources_1125899_36.stderr parsl.dataflow.dflow INFO: Task 3549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db56faa27e936ab312bbd9ba8b24bacf with result from task 3549 parsl.dataflow.dflow DEBUG: Task 3549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/e0427cd6-5316-40eb-8c3a-1892d64ff6b3_isrForCrosstalkSources_1125935_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/e0427cd6-5316-40eb-8c3a-1892d64ff6b3_isrForCrosstalkSources_1125935_44.stderr parsl.dataflow.dflow INFO: Task 3536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b00ad6baf4da22f75cf9717bf6610be4 with result from task 3536 parsl.dataflow.dflow DEBUG: Task 3536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/59b232e9-4d1b-4036-8da7-64741ff03c66_isrForCrosstalkSources_1125934_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/59b232e9-4d1b-4036-8da7-64741ff03c66_isrForCrosstalkSources_1125934_57.stderr parsl.dataflow.dflow INFO: Task 3545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a89c18365ea59fd2d0dc57432e6b00e6 with result from task 3545 parsl.dataflow.dflow DEBUG: Task 3545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/0047afed-ceed-4dad-93c7-5d4b5dfd70d3_isrForCrosstalkSources_1125916_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/0047afed-ceed-4dad-93c7-5d4b5dfd70d3_isrForCrosstalkSources_1125916_48.stderr parsl.dataflow.dflow INFO: Task 3386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc577ae6fef4c748bbcadb9249a2e670 with result from task 3386 parsl.dataflow.dflow DEBUG: Task 3386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8e6350a9-1ec1-40fa-93d7-cc83dbd83504_isrForCrosstalkSources_1125936_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/8e6350a9-1ec1-40fa-93d7-cc83dbd83504_isrForCrosstalkSources_1125936_33.stderr parsl.dataflow.dflow INFO: Task 3590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 210818d45a653b0d132a77e4c249626f with result from task 3590 parsl.dataflow.dflow DEBUG: Task 3590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/ba49e149-8831-46fe-9f72-b79fdd940697_isrForCrosstalkSources_1125924_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/ba49e149-8831-46fe-9f72-b79fdd940697_isrForCrosstalkSources_1125924_8.stderr parsl.dataflow.dflow INFO: Task 3564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0213924d415c96148dc95374fc45ec6 with result from task 3564 parsl.dataflow.dflow DEBUG: Task 3564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/5cf16b00-6b36-4b09-bac4-313d2569e601_isrForCrosstalkSources_1125933_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/5cf16b00-6b36-4b09-bac4-313d2569e601_isrForCrosstalkSources_1125933_15.stderr parsl.dataflow.dflow INFO: Task 3414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2f0ba582bebcb828794297b0dc7a5cf with result from task 3414 parsl.dataflow.dflow DEBUG: Task 3414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/bd19c350-762b-4f58-9659-eb85b00e2249_isrForCrosstalkSources_1125932_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/bd19c350-762b-4f58-9659-eb85b00e2249_isrForCrosstalkSources_1125932_62.stderr parsl.dataflow.dflow INFO: Task 3577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9cdf6379f5b9b951f40f045fe9c8bfb with result from task 3577 parsl.dataflow.dflow DEBUG: Task 3577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/50d0b82d-66a0-427f-89f4-8140a4286a86_isrForCrosstalkSources_1125902_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/50d0b82d-66a0-427f-89f4-8140a4286a86_isrForCrosstalkSources_1125902_58.stderr parsl.dataflow.dflow INFO: Task 3556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70f7bf2e294e78af9241e73ea37f8ce7 with result from task 3556 parsl.dataflow.dflow DEBUG: Task 3556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/f37f8cfd-f38c-4aef-8bc6-3617b9f166a1_isrForCrosstalkSources_1125899_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/f37f8cfd-f38c-4aef-8bc6-3617b9f166a1_isrForCrosstalkSources_1125899_59.stderr parsl.dataflow.dflow INFO: Task 3515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3085e5150884ea21486c117e522a063 with result from task 3515 parsl.dataflow.dflow DEBUG: Task 3515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/901d2002-401f-457a-b9a9-bc990e85f20e_isrForCrosstalkSources_1125939_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/901d2002-401f-457a-b9a9-bc990e85f20e_isrForCrosstalkSources_1125939_3.stderr parsl.dataflow.dflow INFO: Task 3512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27b05f1d0aa61d4472e80d6e1c3d1cd3 with result from task 3512 parsl.dataflow.dflow DEBUG: Task 3512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/09125582-0fab-4cbd-b126-113ee2ed9814_isrForCrosstalkSources_1125936_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/09125582-0fab-4cbd-b126-113ee2ed9814_isrForCrosstalkSources_1125936_50.stderr parsl.dataflow.dflow INFO: Task 3479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75a0441adf9619628a3997779e441008 with result from task 3479 parsl.dataflow.dflow DEBUG: Task 3479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/bf81715b-d484-4594-8fa0-6382693150e5_isrForCrosstalkSources_1125903_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/bf81715b-d484-4594-8fa0-6382693150e5_isrForCrosstalkSources_1125903_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:42 proc_lsst.multi:146 [INFO] found job 16896566 in provider checkpoint proc_lsst.multi INFO: found job 16896566 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a6fcf49fc68ba0ca05bda8550b5b963 with result from task 3402 parsl.dataflow.dflow DEBUG: Task 3402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/a19b3a42-1827-4832-b857-cb3835389b59_isrForCrosstalkSources_1125946_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/a19b3a42-1827-4832-b857-cb3835389b59_isrForCrosstalkSources_1125946_24.stderr parsl.dataflow.dflow INFO: Task 3495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dabc0a2279b15f6c3c0dcac2a4768720 with result from task 3495 parsl.dataflow.dflow DEBUG: Task 3495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/81c99f97-532e-4992-9ce9-32fbb8c9f129_isrForCrosstalkSources_1125938_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/81c99f97-532e-4992-9ce9-32fbb8c9f129_isrForCrosstalkSources_1125938_56.stderr parsl.dataflow.dflow INFO: Task 3403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03798145309629f0bcba64f42b0409c8 with result from task 3403 parsl.dataflow.dflow DEBUG: Task 3403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/90d1f4f2-bcdc-4dd4-a94c-f8ae498a4e51_isrForCrosstalkSources_1125939_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/90d1f4f2-bcdc-4dd4-a94c-f8ae498a4e51_isrForCrosstalkSources_1125939_24.stderr parsl.dataflow.dflow INFO: Task 3522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddd3712fe9150e4090efa4450a5c305b with result from task 3522 parsl.dataflow.dflow DEBUG: Task 3522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/04eebbb6-cd61-4835-8db3-6cf711689c63_isrForCrosstalkSources_1125901_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/04eebbb6-cd61-4835-8db3-6cf711689c63_isrForCrosstalkSources_1125901_52.stderr parsl.dataflow.dflow INFO: Task 3531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1ec431cd40c7c46260e72458618375b with result from task 3531 parsl.dataflow.dflow DEBUG: Task 3531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/75878e58-d83c-43c4-b216-86432ea07fe7_isrForCrosstalkSources_1125936_20.stdout parsl.dataflow.dflow INFO: Standard error for task 3531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/75878e58-d83c-43c4-b216-86432ea07fe7_isrForCrosstalkSources_1125936_20.stderr parsl.dataflow.dflow INFO: Task 3490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 766832efa44c30ad27824217ad5704d6 with result from task 3490 parsl.dataflow.dflow DEBUG: Task 3490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/7b3afb71-4f78-4830-be62-7bc959c47c34_isrForCrosstalkSources_1125937_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/7b3afb71-4f78-4830-be62-7bc959c47c34_isrForCrosstalkSources_1125937_47.stderr parsl.dataflow.dflow INFO: Task 3469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d4f38444523ab47dbe0ea4e84e1f196 with result from task 3469 parsl.dataflow.dflow DEBUG: Task 3469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/3a90de7a-cf40-4860-bcf1-207176fafa8f_isrForCrosstalkSources_1125905_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/3a90de7a-cf40-4860-bcf1-207176fafa8f_isrForCrosstalkSources_1125905_62.stderr parsl.dataflow.dflow INFO: Task 3585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a90eb4ce75b2791fcb3baf054c01edb with result from task 3585 parsl.dataflow.dflow DEBUG: Task 3585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/0d50e03b-65a4-4b72-874c-b7d26d326316_isrForCrosstalkSources_1125900_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/0d50e03b-65a4-4b72-874c-b7d26d326316_isrForCrosstalkSources_1125900_39.stderr parsl.dataflow.dflow INFO: Task 3588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8bef1b7f978e8aa79060c8be062c166 with result from task 3588 parsl.dataflow.dflow DEBUG: Task 3588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/98d0a7e7-5604-4438-a3a7-ef0226e9dcc5_isrForCrosstalkSources_1125923_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/98d0a7e7-5604-4438-a3a7-ef0226e9dcc5_isrForCrosstalkSources_1125923_33.stderr parsl.dataflow.dflow INFO: Task 3452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fc7be7ecf15ee0b0599ed69002d0dc9 with result from task 3452 parsl.dataflow.dflow DEBUG: Task 3452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/410e264d-3a19-4d68-95ac-7a1843dd122e_isrForCrosstalkSources_1125938_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/410e264d-3a19-4d68-95ac-7a1843dd122e_isrForCrosstalkSources_1125938_36.stderr parsl.dataflow.dflow INFO: Task 3533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85d2dfaea4e132b543c0f5ce2d552123 with result from task 3533 parsl.dataflow.dflow DEBUG: Task 3533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/bf92a2ec-1d71-442e-81e1-33918adc5c85_isrForCrosstalkSources_1125935_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/bf92a2ec-1d71-442e-81e1-33918adc5c85_isrForCrosstalkSources_1125935_43.stderr parsl.dataflow.dflow INFO: Task 3521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry beb4ae1e083ddedf5a275d214a1272c6 with result from task 3521 parsl.dataflow.dflow DEBUG: Task 3521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/41815a71-1776-4fb2-86e1-3ad3abfca06a_isrForCrosstalkSources_1125944_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/41815a71-1776-4fb2-86e1-3ad3abfca06a_isrForCrosstalkSources_1125944_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:42 proc_lsst.multi:146 [INFO] found job 16896568 in provider checkpoint proc_lsst.multi INFO: found job 16896568 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 574f6d2150a8a6d42223d4c07d01d933 with result from task 3607 parsl.dataflow.dflow DEBUG: Task 3607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/22456a59-6c79-4e7e-b31d-009831fb0ad9_isrForCrosstalkSources_1125889_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/22456a59-6c79-4e7e-b31d-009831fb0ad9_isrForCrosstalkSources_1125889_50.stderr parsl.dataflow.dflow INFO: Task 3576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2b323e2f65b4ee1cf2291db555c287a with result from task 3576 parsl.dataflow.dflow DEBUG: Task 3576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/925326ae-7c3a-427c-984b-7f3e54498a88_isrForCrosstalkSources_1125913_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/925326ae-7c3a-427c-984b-7f3e54498a88_isrForCrosstalkSources_1125913_48.stderr parsl.dataflow.dflow INFO: Task 3413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b19eb247f9f2ad737b916c50b35caaf with result from task 3413 parsl.dataflow.dflow DEBUG: Task 3413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a0ac7aa8-18d0-4850-aad1-34315eee3a8d_isrForCrosstalkSources_1125921_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125921/a0ac7aa8-18d0-4850-aad1-34315eee3a8d_isrForCrosstalkSources_1125921_6.stderr parsl.dataflow.dflow INFO: Task 3602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3aac311a1518b9f32144da5dd0e25667 with result from task 3602 parsl.dataflow.dflow DEBUG: Task 3602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/80f3fc3a-d3fb-4f35-aff8-bb5f59846500_isrForCrosstalkSources_1125945_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/80f3fc3a-d3fb-4f35-aff8-bb5f59846500_isrForCrosstalkSources_1125945_29.stderr parsl.dataflow.dflow INFO: Task 3439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 812e5371d662127d2549b1597012fbb0 with result from task 3439 parsl.dataflow.dflow DEBUG: Task 3439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/6aed66e4-5bfa-44c7-bfd1-6dc8757ed714_isrForCrosstalkSources_1125923_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/6aed66e4-5bfa-44c7-bfd1-6dc8757ed714_isrForCrosstalkSources_1125923_2.stderr parsl.dataflow.dflow INFO: Task 3582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f32ee143ef8627863b10c4061ccf8a1e with result from task 3582 parsl.dataflow.dflow DEBUG: Task 3582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/6b1f4d88-4aca-4c7b-bae9-b9e61f5e1aac_isrForCrosstalkSources_1125930_52.stdout parsl.dataflow.dflow INFO: Standard error for task 3582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/6b1f4d88-4aca-4c7b-bae9-b9e61f5e1aac_isrForCrosstalkSources_1125930_52.stderr parsl.dataflow.dflow INFO: Task 3543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ddc89515ab64389af48dc06354c617c with result from task 3543 parsl.dataflow.dflow DEBUG: Task 3543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/88e9fae8-dfd0-4a18-bfa2-60e8d0ed0d55_isrForCrosstalkSources_1125944_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/88e9fae8-dfd0-4a18-bfa2-60e8d0ed0d55_isrForCrosstalkSources_1125944_4.stderr parsl.dataflow.dflow INFO: Task 3615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f19e6c89149dcae90fc35b4cf84ab777 with result from task 3615 parsl.dataflow.dflow DEBUG: Task 3615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/fe50e467-2315-4593-ab4a-bbaa7117b20f_isrForCrosstalkSources_1125918_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/fe50e467-2315-4593-ab4a-bbaa7117b20f_isrForCrosstalkSources_1125918_16.stderr parsl.dataflow.dflow INFO: Task 3575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca7bd32bb35bb01c78e04426fb345d65 with result from task 3575 parsl.dataflow.dflow DEBUG: Task 3575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c5ed509d-d7b2-4d53-8475-820ecb0a33c1_isrForCrosstalkSources_1125917_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/c5ed509d-d7b2-4d53-8475-820ecb0a33c1_isrForCrosstalkSources_1125917_32.stderr parsl.dataflow.dflow INFO: Task 3600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc8b6954f4f034945d407c91454ca30a with result from task 3600 parsl.dataflow.dflow DEBUG: Task 3600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/b5c2dff2-74ae-403a-a3e9-323840277cc7_isrForCrosstalkSources_1125898_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/b5c2dff2-74ae-403a-a3e9-323840277cc7_isrForCrosstalkSources_1125898_48.stderr parsl.dataflow.dflow INFO: Task 3552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bf119df5fea8a779cdd2145e4e12158 with result from task 3552 parsl.dataflow.dflow DEBUG: Task 3552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/b24537e5-0d92-44d4-aead-25298829177c_isrForCrosstalkSources_1125895_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125895/b24537e5-0d92-44d4-aead-25298829177c_isrForCrosstalkSources_1125895_11.stderr parsl.dataflow.dflow INFO: Task 3599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e3abe0daf3ea12efa71b387fb218536 with result from task 3599 parsl.dataflow.dflow DEBUG: Task 3599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/b5cd5194-e348-448a-9ecc-2b2017047a46_isrForCrosstalkSources_1125897_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/b5cd5194-e348-448a-9ecc-2b2017047a46_isrForCrosstalkSources_1125897_5.stderr parsl.dataflow.dflow INFO: Task 3532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a60b3b13458c2af097079be1673c1f84 with result from task 3532 parsl.dataflow.dflow DEBUG: Task 3532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/d9e75b79-f25e-4756-ad45-61a1928fe33f_isrForCrosstalkSources_1125904_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/d9e75b79-f25e-4756-ad45-61a1928fe33f_isrForCrosstalkSources_1125904_7.stderr parsl.dataflow.dflow INFO: Task 3527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d2f014890efabe8338ebdf76c3c00de with result from task 3527 parsl.dataflow.dflow DEBUG: Task 3527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/957f0257-fd43-441d-b610-cc4f08f71d08_isrForCrosstalkSources_1125946_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/957f0257-fd43-441d-b610-cc4f08f71d08_isrForCrosstalkSources_1125946_49.stderr parsl.dataflow.dflow INFO: Task 3476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12a3409000daa0d55d6576cca598b67c with result from task 3476 parsl.dataflow.dflow DEBUG: Task 3476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/ae42c24a-e26b-420c-a1aa-bf5496ffcd22_isrForCrosstalkSources_1125888_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/ae42c24a-e26b-420c-a1aa-bf5496ffcd22_isrForCrosstalkSources_1125888_42.stderr parsl.dataflow.dflow INFO: Task 3586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99f41c7aa0666fb46619ea283526bb5b with result from task 3586 parsl.dataflow.dflow DEBUG: Task 3586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/b906ca29-4406-4b19-b627-49883510e5b3_isrForCrosstalkSources_1125943_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/b906ca29-4406-4b19-b627-49883510e5b3_isrForCrosstalkSources_1125943_2.stderr parsl.dataflow.dflow INFO: Task 3580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d17fd6f928b0a5e33f3af5009ece26fa with result from task 3580 parsl.dataflow.dflow DEBUG: Task 3580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d094d585-ae52-4110-96ed-2df934b315e9_isrForCrosstalkSources_1125915_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d094d585-ae52-4110-96ed-2df934b315e9_isrForCrosstalkSources_1125915_2.stderr parsl.dataflow.dflow INFO: Task 3604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38db72af569e6bf971303e3831adddd8 with result from task 3604 parsl.dataflow.dflow DEBUG: Task 3604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/0afe984d-8117-476f-a6bd-af6736c4e354_isrForCrosstalkSources_1125919_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/0afe984d-8117-476f-a6bd-af6736c4e354_isrForCrosstalkSources_1125919_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:43 proc_lsst.multi:146 [INFO] found job 16896569 in provider checkpoint proc_lsst.multi INFO: found job 16896569 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d66f43dc069d627b897b70f6ebd0026a with result from task 3464 parsl.dataflow.dflow DEBUG: Task 3464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/068df873-d6a0-492e-8934-296752879429_isrForCrosstalkSources_1125935_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/068df873-d6a0-492e-8934-296752879429_isrForCrosstalkSources_1125935_57.stderr parsl.dataflow.dflow INFO: Task 3633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ea6defd3dbf013522d89cdcb3087692 with result from task 3633 parsl.dataflow.dflow DEBUG: Task 3633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d5114e4b-7905-462e-9579-80dd187b6d72_isrForCrosstalkSources_1125940_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d5114e4b-7905-462e-9579-80dd187b6d72_isrForCrosstalkSources_1125940_26.stderr parsl.dataflow.dflow INFO: Task 3534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5449d278f657e67fbf294d82a9676d90 with result from task 3534 parsl.dataflow.dflow DEBUG: Task 3534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e847c49f-ab6e-4458-8504-a26f26495b2f_isrForCrosstalkSources_1125937_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/e847c49f-ab6e-4458-8504-a26f26495b2f_isrForCrosstalkSources_1125937_46.stderr parsl.dataflow.dflow INFO: Task 3614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1eef6b3eb2bb2ec7618241735854c2e1 with result from task 3614 parsl.dataflow.dflow DEBUG: Task 3614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/d6847986-a66c-4be6-b7b7-3eaf42cf9f7b_isrForCrosstalkSources_1125913_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/d6847986-a66c-4be6-b7b7-3eaf42cf9f7b_isrForCrosstalkSources_1125913_14.stderr parsl.dataflow.dflow INFO: Task 3535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bf4676b93634cdaa508e7f4b04ad114 with result from task 3535 parsl.dataflow.dflow DEBUG: Task 3535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/709c70f4-2a25-45d8-b95d-5b60bc24cf4f_isrForCrosstalkSources_1125917_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/709c70f4-2a25-45d8-b95d-5b60bc24cf4f_isrForCrosstalkSources_1125917_14.stderr parsl.dataflow.dflow INFO: Task 3505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b169e08216219a2b1fc9d05f80e9dc3 with result from task 3505 parsl.dataflow.dflow DEBUG: Task 3505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/2f2a9da4-9c23-47aa-bd26-6ad0320a2094_isrForCrosstalkSources_1125923_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/2f2a9da4-9c23-47aa-bd26-6ad0320a2094_isrForCrosstalkSources_1125923_21.stderr parsl.dataflow.dflow INFO: Task 3589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48e8ab548939ddc027d653ff606e82f5 with result from task 3589 parsl.dataflow.dflow DEBUG: Task 3589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/5111c325-bdcb-4a55-a285-972ae3b10bee_isrForCrosstalkSources_1125894_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/5111c325-bdcb-4a55-a285-972ae3b10bee_isrForCrosstalkSources_1125894_15.stderr parsl.dataflow.dflow INFO: Task 3528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d129bb39ecc2a4008f427c8e84b9402b with result from task 3528 parsl.dataflow.dflow DEBUG: Task 3528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/c61caa0d-2356-435d-acda-6867d5282301_isrForCrosstalkSources_1125946_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/c61caa0d-2356-435d-acda-6867d5282301_isrForCrosstalkSources_1125946_51.stderr parsl.dataflow.dflow INFO: Task 3619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4d23728f22860bd1db04b4599e1e331 with result from task 3619 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:43 proc_lsst.multi:146 [INFO] found job 16896570 in provider checkpoint proc_lsst.multi INFO: found job 16896570 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 3619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/a6857d9c-e50f-441e-92e4-4ead25991dd6_isrForCrosstalkSources_1125942_46.stdout parsl.dataflow.dflow INFO: Standard error for task 3619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/a6857d9c-e50f-441e-92e4-4ead25991dd6_isrForCrosstalkSources_1125942_46.stderr parsl.dataflow.dflow INFO: Task 3438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 507bd88a8094036cfe256eddbde000fd with result from task 3438 parsl.dataflow.dflow DEBUG: Task 3438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/48c0fad6-8b86-4a98-bd3f-709005b6288f_isrForCrosstalkSources_1125902_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/48c0fad6-8b86-4a98-bd3f-709005b6288f_isrForCrosstalkSources_1125902_31.stderr parsl.dataflow.dflow INFO: Task 3555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dfa73b638402d94ef8a683c0336e237 with result from task 3555 parsl.dataflow.dflow DEBUG: Task 3555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/d13f2e06-8d57-4dd2-85cb-d42700144f91_isrForCrosstalkSources_1125922_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/d13f2e06-8d57-4dd2-85cb-d42700144f91_isrForCrosstalkSources_1125922_9.stderr parsl.dataflow.dflow INFO: Task 3567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba22f9b832001e1a78a631c5a73c7f7c with result from task 3567 parsl.dataflow.dflow DEBUG: Task 3567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/7b6e9d98-5833-473d-9856-9ed0606e6466_isrForCrosstalkSources_1125894_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/7b6e9d98-5833-473d-9856-9ed0606e6466_isrForCrosstalkSources_1125894_29.stderr parsl.dataflow.dflow INFO: Task 3540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b384dd5d701ae2ffe7cbde1f22e26d7 with result from task 3540 parsl.dataflow.dflow DEBUG: Task 3540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/f2a529c5-410c-409b-91e1-f0ab5dc2191f_isrForCrosstalkSources_1125911_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/f2a529c5-410c-409b-91e1-f0ab5dc2191f_isrForCrosstalkSources_1125911_23.stderr parsl.dataflow.dflow INFO: Task 3557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b398ad817b7b2933c7508cd016bc105 with result from task 3557 parsl.dataflow.dflow DEBUG: Task 3557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/f30ab3e5-6bf4-4bed-acd6-d7f5f49e7eb2_isrForCrosstalkSources_1125934_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125934/f30ab3e5-6bf4-4bed-acd6-d7f5f49e7eb2_isrForCrosstalkSources_1125934_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:43 proc_lsst.multi:146 [INFO] found job 16896571 in provider checkpoint proc_lsst.multi INFO: found job 16896571 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a7704e4671fda90184093a373b6f732 with result from task 3519 parsl.dataflow.dflow DEBUG: Task 3519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/a2f568d4-27af-4efe-b4fb-6779069745ef_isrForCrosstalkSources_1125892_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/a2f568d4-27af-4efe-b4fb-6779069745ef_isrForCrosstalkSources_1125892_12.stderr parsl.dataflow.dflow INFO: Task 3548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37e2f617c58667baf1e7e44a4aa79b87 with result from task 3548 parsl.dataflow.dflow DEBUG: Task 3548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/6d31d861-bcbc-48bc-a087-d925afd4d847_isrForCrosstalkSources_1125925_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/6d31d861-bcbc-48bc-a087-d925afd4d847_isrForCrosstalkSources_1125925_27.stderr parsl.dataflow.dflow INFO: Task 3579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70ae99036c8b235e658fa01f63f9cbcb with result from task 3579 parsl.dataflow.dflow DEBUG: Task 3579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/2c8b40ce-e25e-4e10-aa20-ccb621806e0b_isrForCrosstalkSources_1125897_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/2c8b40ce-e25e-4e10-aa20-ccb621806e0b_isrForCrosstalkSources_1125897_15.stderr parsl.dataflow.dflow INFO: Task 3538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42e2a6074f8e3670e14efd26222a8ddf with result from task 3538 parsl.dataflow.dflow DEBUG: Task 3538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/948a95c6-ee36-4acd-8152-ce6c774427a5_isrForCrosstalkSources_1125935_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/948a95c6-ee36-4acd-8152-ce6c774427a5_isrForCrosstalkSources_1125935_54.stderr parsl.dataflow.dflow INFO: Task 3561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2655a8b2ec6e91a49921718754b221fb with result from task 3561 parsl.dataflow.dflow DEBUG: Task 3561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/06e8ac17-ff06-4a57-aa18-c017fe5caba2_isrForCrosstalkSources_1125897_32.stdout parsl.dataflow.dflow INFO: Standard error for task 3561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/06e8ac17-ff06-4a57-aa18-c017fe5caba2_isrForCrosstalkSources_1125897_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3587 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 94d806518cc849d1fd113523b3682077 with result from task 3587 parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3587 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/b94b8344-6eb5-40e4-862a-9aa04207d75d_isrForCrosstalkSources_1125892_9.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/b94b8344-6eb5-40e4-862a-9aa04207d75d_isrForCrosstalkSources_1125892_9.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:44 proc_lsst.multi:146 [INFO] found job 16896572 in provider checkpoint proc_lsst.multi INFO: found job 16896572 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0d3773c31e82374ec4bf7e187586b5b with result from task 3554 parsl.dataflow.dflow DEBUG: Task 3554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/9b1213da-7d8c-4110-bd88-571c0c56f5ab_isrForCrosstalkSources_1125891_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/9b1213da-7d8c-4110-bd88-571c0c56f5ab_isrForCrosstalkSources_1125891_38.stderr parsl.dataflow.dflow INFO: Task 3558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53a2df01a782d092e28e296f0ab02fd5 with result from task 3558 parsl.dataflow.dflow DEBUG: Task 3558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/5e6b9290-7ef5-4a84-8116-bf3b9279aa21_isrForCrosstalkSources_1125932_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/5e6b9290-7ef5-4a84-8116-bf3b9279aa21_isrForCrosstalkSources_1125932_14.stderr parsl.dataflow.dflow INFO: Task 3640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 276a67f2223c8f327a85405339f6dc24 with result from task 3640 parsl.dataflow.dflow DEBUG: Task 3640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/29b7c81b-9855-4faf-b976-fd902394c1a6_isrForCrosstalkSources_1125916_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/29b7c81b-9855-4faf-b976-fd902394c1a6_isrForCrosstalkSources_1125916_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:44 proc_lsst.multi:146 [INFO] found job 16896573 in provider checkpoint proc_lsst.multi INFO: found job 16896573 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0e43ba0709d29335924658af22f8de0 with result from task 3480 parsl.dataflow.dflow DEBUG: Task 3480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/7f15679d-2296-4f5c-a0c1-90993a5622b5_isrForCrosstalkSources_1125932_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/7f15679d-2296-4f5c-a0c1-90993a5622b5_isrForCrosstalkSources_1125932_11.stderr parsl.dataflow.dflow INFO: Task 3525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 729c37b175ec9830f92b7456f4a3bcf7 with result from task 3525 parsl.dataflow.dflow DEBUG: Task 3525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/76074ff3-913a-4c38-8125-10a39c6dbc37_isrForCrosstalkSources_1125889_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/76074ff3-913a-4c38-8125-10a39c6dbc37_isrForCrosstalkSources_1125889_5.stderr parsl.dataflow.dflow INFO: Task 3491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee64bb20c0dabde3800a12922bb2c131 with result from task 3491 parsl.dataflow.dflow DEBUG: Task 3491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/9ecd4a37-f5b3-4fe5-b880-b8a91289be84_isrForCrosstalkSources_1125929_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/9ecd4a37-f5b3-4fe5-b880-b8a91289be84_isrForCrosstalkSources_1125929_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:44 proc_lsst.multi:146 [INFO] found job 16896574 in provider checkpoint proc_lsst.multi INFO: found job 16896574 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acc20d93b7a9336ebb67ef3a8bb3f89b with result from task 3571 parsl.dataflow.dflow DEBUG: Task 3571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/21b8b7ff-8605-44ba-b115-a7235d28d8a4_isrForCrosstalkSources_1125929_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/21b8b7ff-8605-44ba-b115-a7235d28d8a4_isrForCrosstalkSources_1125929_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3592 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 9005e8b8f9f91e09fbdd49db9de528f4 with result from task 3592 parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3592 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6c83a70d-d4e5-4756-932b-c706be95a3ef_isrForCrosstalkSources_1125909_56.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/6c83a70d-d4e5-4756-932b-c706be95a3ef_isrForCrosstalkSources_1125909_56.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3493 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 301d02d5886ea67622ef97661ac69ad8 with result from task 3493 parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3493 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/b65b59cd-2cd6-4719-b8dd-e00dd4a4c4dd_isrForCrosstalkSources_1125935_35.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/b65b59cd-2cd6-4719-b8dd-e00dd4a4c4dd_isrForCrosstalkSources_1125935_35.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:44 proc_lsst.multi:146 [INFO] found job 16896575 in provider checkpoint proc_lsst.multi INFO: found job 16896575 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61674e57c8af276465ea4546da1dcdcf with result from task 3435 parsl.dataflow.dflow DEBUG: Task 3435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/3724a779-46fc-4743-8369-db2785d6633a_isrForCrosstalkSources_1125929_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/3724a779-46fc-4743-8369-db2785d6633a_isrForCrosstalkSources_1125929_48.stderr parsl.dataflow.dflow INFO: Task 3560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed827a980cb5e7813c04be2ae2692308 with result from task 3560 parsl.dataflow.dflow DEBUG: Task 3560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/d18ea9e4-c86a-4aeb-bd1d-07d43395ce44_isrForCrosstalkSources_1125938_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125938/d18ea9e4-c86a-4aeb-bd1d-07d43395ce44_isrForCrosstalkSources_1125938_43.stderr parsl.dataflow.dflow INFO: Task 3483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1026fe55b47264a398e4ee07a0edc99f with result from task 3483 parsl.dataflow.dflow DEBUG: Task 3483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/2b7d5649-5da8-4eaf-ae27-d486df580839_isrForCrosstalkSources_1125911_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/2b7d5649-5da8-4eaf-ae27-d486df580839_isrForCrosstalkSources_1125911_53.stderr parsl.dataflow.dflow INFO: Task 3471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8ba2fb6e22f01dd8a452b9b75a384c3 with result from task 3471 parsl.dataflow.dflow DEBUG: Task 3471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d6af6a82-9c47-4f52-b789-aa5917c842da_isrForCrosstalkSources_1125919_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d6af6a82-9c47-4f52-b789-aa5917c842da_isrForCrosstalkSources_1125919_37.stderr parsl.dataflow.dflow INFO: Task 3630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be8e41c7702ef99f80c107b2325edc7d with result from task 3630 parsl.dataflow.dflow DEBUG: Task 3630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/ebed7e01-54de-45e7-87c0-79b830c08f9d_isrForCrosstalkSources_1125924_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125924/ebed7e01-54de-45e7-87c0-79b830c08f9d_isrForCrosstalkSources_1125924_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3463 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry ffa5d44d376f239eec8799dfddaf9a82 with result from task 3463 parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3463 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/52ebfd9b-8dc8-4e2f-9e55-d7169d212c04_isrForCrosstalkSources_1125945_49.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/52ebfd9b-8dc8-4e2f-9e55-d7169d212c04_isrForCrosstalkSources_1125945_49.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:45 proc_lsst.multi:146 [INFO] found job 16896576 in provider checkpoint proc_lsst.multi INFO: found job 16896576 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b42891667f6f02c9df30c0ed56b0e17 with result from task 3494 parsl.dataflow.dflow DEBUG: Task 3494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a501d414-2836-4233-beb5-d98c911fec16_isrForCrosstalkSources_1125945_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a501d414-2836-4233-beb5-d98c911fec16_isrForCrosstalkSources_1125945_9.stderr parsl.dataflow.dflow INFO: Task 3506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f24c04e46b8b9e6e5883bd986c5d3b7f with result from task 3506 parsl.dataflow.dflow DEBUG: Task 3506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/b38547ba-0166-4416-b412-152c340a1d16_isrForCrosstalkSources_1125942_54.stdout parsl.dataflow.dflow INFO: Standard error for task 3506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/b38547ba-0166-4416-b412-152c340a1d16_isrForCrosstalkSources_1125942_54.stderr parsl.dataflow.dflow INFO: Task 3478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a456ccdf5c65a0f3a6bc4a5ee1ec7226 with result from task 3478 parsl.dataflow.dflow DEBUG: Task 3478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3491284c-d3ab-4492-9c1d-c1dd7133b916_isrForCrosstalkSources_1125941_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/3491284c-d3ab-4492-9c1d-c1dd7133b916_isrForCrosstalkSources_1125941_48.stderr parsl.dataflow.dflow INFO: Task 3529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acd41092dcb4f4d848d5939a8382fdb7 with result from task 3529 parsl.dataflow.dflow DEBUG: Task 3529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/eab24858-6666-45bf-b1d2-94a8387e6feb_isrForCrosstalkSources_1125943_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/eab24858-6666-45bf-b1d2-94a8387e6feb_isrForCrosstalkSources_1125943_47.stderr parsl.dataflow.dflow INFO: Task 3444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ab964b3c3db74d0e2da75734bb6fad4 with result from task 3444 parsl.dataflow.dflow DEBUG: Task 3444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/101157cd-5eea-4a0d-a81f-68980040cd05_isrForCrosstalkSources_1125941_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/101157cd-5eea-4a0d-a81f-68980040cd05_isrForCrosstalkSources_1125941_21.stderr parsl.dataflow.dflow INFO: Task 3492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc64f550f8846d1947f508ccce0a007c with result from task 3492 parsl.dataflow.dflow DEBUG: Task 3492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/7d618e8d-bb51-497d-9092-55b7a40bd1ce_isrForCrosstalkSources_1125928_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/7d618e8d-bb51-497d-9092-55b7a40bd1ce_isrForCrosstalkSources_1125928_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:45 proc_lsst.multi:146 [INFO] found job 16896577 in provider checkpoint proc_lsst.multi INFO: found job 16896577 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3610 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry f5e32888c2995a67935a8a41afd0220e with result from task 3610 parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3610 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/a2da7c12-c315-413d-8222-db0f6b9954c7_isrForCrosstalkSources_1125929_37.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/a2da7c12-c315-413d-8222-db0f6b9954c7_isrForCrosstalkSources_1125929_37.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:45 proc_lsst.multi:146 [INFO] found job 16896578 in provider checkpoint proc_lsst.multi INFO: found job 16896578 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bbe80de6f2f9961f9d4eb78c4a0e854 with result from task 3639 parsl.dataflow.dflow DEBUG: Task 3639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/86955407-6ac5-4181-aecf-93c37572f2f6_isrForCrosstalkSources_1125939_51.stdout parsl.dataflow.dflow INFO: Standard error for task 3639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/86955407-6ac5-4181-aecf-93c37572f2f6_isrForCrosstalkSources_1125939_51.stderr parsl.dataflow.dflow INFO: Task 3605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e67cc9949e1cdd2f16f7ffcb8c5913cc with result from task 3605 parsl.dataflow.dflow DEBUG: Task 3605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/c67ddf6f-4825-4081-b04b-21a46fda9a96_isrForCrosstalkSources_1125898_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/c67ddf6f-4825-4081-b04b-21a46fda9a96_isrForCrosstalkSources_1125898_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:45 proc_lsst.multi:146 [INFO] found job 16896579 in provider checkpoint proc_lsst.multi INFO: found job 16896579 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d14b440fad839c54825b88190e6c27c with result from task 3603 parsl.dataflow.dflow DEBUG: Task 3603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/deb427ff-27be-4e77-8abc-5ef6d24b0adf_isrForCrosstalkSources_1125933_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/deb427ff-27be-4e77-8abc-5ef6d24b0adf_isrForCrosstalkSources_1125933_10.stderr parsl.dataflow.dflow INFO: Task 3628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5813754b6a6435f0f167326d5503b52 with result from task 3628 parsl.dataflow.dflow DEBUG: Task 3628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/c8e06794-95b9-4cf2-996b-e3fa2890d74a_isrForCrosstalkSources_1125940_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/c8e06794-95b9-4cf2-996b-e3fa2890d74a_isrForCrosstalkSources_1125940_16.stderr parsl.dataflow.dflow INFO: Task 3581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4eb75f78008682ef67633540af1c1b7d with result from task 3581 parsl.dataflow.dflow DEBUG: Task 3581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/ab584911-65ae-4997-94d2-776c20b2b1bb_isrForCrosstalkSources_1125890_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/ab584911-65ae-4997-94d2-776c20b2b1bb_isrForCrosstalkSources_1125890_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:45 proc_lsst.multi:146 [INFO] found job 16896580 in provider checkpoint proc_lsst.multi INFO: found job 16896580 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4241df9a2c4153c2a453a6309a594c2 with result from task 3626 parsl.dataflow.dflow DEBUG: Task 3626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/4e48b53f-df0c-4d8d-8815-a0826550928d_isrForCrosstalkSources_1125915_27.stdout parsl.dataflow.dflow INFO: Standard error for task 3626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/4e48b53f-df0c-4d8d-8815-a0826550928d_isrForCrosstalkSources_1125915_27.stderr parsl.dataflow.dflow INFO: Task 3583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89e6b54681c10d17d1fff0e7b8cd78fd with result from task 3583 parsl.dataflow.dflow DEBUG: Task 3583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/9e3a368f-405f-494a-bf8b-f5627703b427_isrForCrosstalkSources_1125896_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/9e3a368f-405f-494a-bf8b-f5627703b427_isrForCrosstalkSources_1125896_41.stderr parsl.dataflow.dflow INFO: Task 3559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cad454c8fb9a78e238b48de25eea10eb with result from task 3559 parsl.dataflow.dflow DEBUG: Task 3559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/d65fba4a-4b82-41f5-85a4-399e19640e7d_isrForCrosstalkSources_1125896_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125896/d65fba4a-4b82-41f5-85a4-399e19640e7d_isrForCrosstalkSources_1125896_24.stderr parsl.dataflow.dflow INFO: Task 3570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83d8ef8ca161ea4d6c3886b380036c73 with result from task 3570 parsl.dataflow.dflow DEBUG: Task 3570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/146d0270-b61e-4857-83e4-0074b88d7bbe_isrForCrosstalkSources_1125927_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/146d0270-b61e-4857-83e4-0074b88d7bbe_isrForCrosstalkSources_1125927_38.stderr parsl.dataflow.dflow INFO: Task 3623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f78195ac5fb9a6a7e74d0f161ea91d1d with result from task 3623 parsl.dataflow.dflow DEBUG: Task 3623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/70a7aed6-c7d5-4498-aef8-bb566152ba26_isrForCrosstalkSources_1125931_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/70a7aed6-c7d5-4498-aef8-bb566152ba26_isrForCrosstalkSources_1125931_49.stderr parsl.dataflow.dflow INFO: Task 3611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cca2ba5030fe2ddfaa671175f176037 with result from task 3611 parsl.dataflow.dflow DEBUG: Task 3611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/23eeeda8-4bf2-44a2-aa4f-88d40ead79bd_isrForCrosstalkSources_1125935_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/23eeeda8-4bf2-44a2-aa4f-88d40ead79bd_isrForCrosstalkSources_1125935_25.stderr parsl.dataflow.dflow INFO: Task 3635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab273535f9e40f3365e4a628735a8b81 with result from task 3635 parsl.dataflow.dflow DEBUG: Task 3635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/79f9f2f0-babb-4e81-a09d-b1f77523cafa_isrForCrosstalkSources_1125913_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/79f9f2f0-babb-4e81-a09d-b1f77523cafa_isrForCrosstalkSources_1125913_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:46 proc_lsst.multi:146 [INFO] found job 16896581 in provider checkpoint proc_lsst.multi INFO: found job 16896581 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7eddbe1d8ec3bc7c5b873a9b7951c8b4 with result from task 3657 parsl.dataflow.dflow DEBUG: Task 3657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/32c64b8a-7b4c-4df0-8e51-af73c3cc6fda_isrForCrosstalkSources_1125893_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125893/32c64b8a-7b4c-4df0-8e51-af73c3cc6fda_isrForCrosstalkSources_1125893_45.stderr parsl.dataflow.dflow INFO: Task 3566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecd1c1feba6462c2a4776a860e49a672 with result from task 3566 parsl.dataflow.dflow DEBUG: Task 3566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/fbe01cff-bb00-4265-9ec0-69a17d304b03_isrForCrosstalkSources_1125892_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/fbe01cff-bb00-4265-9ec0-69a17d304b03_isrForCrosstalkSources_1125892_62.stderr parsl.dataflow.dflow INFO: Task 3648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd597d238ddc57874f6969942f1ed50e with result from task 3648 parsl.dataflow.dflow DEBUG: Task 3648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/e4ebe6fa-c0c9-4540-94b3-19474f8bb510_isrForCrosstalkSources_1125898_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/e4ebe6fa-c0c9-4540-94b3-19474f8bb510_isrForCrosstalkSources_1125898_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3565 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 2a6113716da55b19ad651063d075fdfe with result from task 3565 parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3565 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/385b5a8f-ae5b-42f7-aedc-734681e7e69f_isrForCrosstalkSources_1125918_25.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/385b5a8f-ae5b-42f7-aedc-734681e7e69f_isrForCrosstalkSources_1125918_25.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3634 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 1a28d91a5fe283e9d160a3c548afbb83 with result from task 3634 parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3634 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/ce518090-f2d7-42a5-a8f2-cee7c58269c2_isrForCrosstalkSources_1125911_40.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/ce518090-f2d7-42a5-a8f2-cee7c58269c2_isrForCrosstalkSources_1125911_40.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:46 proc_lsst.multi:146 [INFO] found job 16896582 in provider checkpoint proc_lsst.multi INFO: found job 16896582 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e02999bbd1cf9b7a028eaebafbccce2 with result from task 3601 parsl.dataflow.dflow DEBUG: Task 3601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f3e7a046-fb61-451a-99af-df1e661355bb_isrForCrosstalkSources_1125922_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/f3e7a046-fb61-451a-99af-df1e661355bb_isrForCrosstalkSources_1125922_3.stderr parsl.dataflow.dflow INFO: Task 3616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11b98d981f16b53ff968c4dea89e2f10 with result from task 3616 parsl.dataflow.dflow DEBUG: Task 3616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/eebe45a1-37fb-4b4d-87d9-502080a38445_isrForCrosstalkSources_1125930_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/eebe45a1-37fb-4b4d-87d9-502080a38445_isrForCrosstalkSources_1125930_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3638 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry cf232f992c5d8397f9dc14412c64e58a with result from task 3638 parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3638 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/9b951d20-9266-479d-9d01-0573edc46215_isrForCrosstalkSources_1125916_15.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/9b951d20-9266-479d-9d01-0573edc46215_isrForCrosstalkSources_1125916_15.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:46 proc_lsst.multi:146 [INFO] found job 16896583 in provider checkpoint proc_lsst.multi INFO: found job 16896583 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 460d8f81cbcc9d545c99233d3d3f1a27 with result from task 3595 parsl.dataflow.dflow DEBUG: Task 3595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d07d3114-bf6b-4668-b6bc-b79f19068b49_isrForCrosstalkSources_1125940_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/d07d3114-bf6b-4668-b6bc-b79f19068b49_isrForCrosstalkSources_1125940_4.stderr parsl.dataflow.dflow INFO: Task 3613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 895aa75dd6f509b31fba1e067f4902aa with result from task 3613 parsl.dataflow.dflow DEBUG: Task 3613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/e9c2ecea-afc9-44d9-a1bc-ca53690777d8_isrForCrosstalkSources_1125903_45.stdout parsl.dataflow.dflow INFO: Standard error for task 3613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125903/e9c2ecea-afc9-44d9-a1bc-ca53690777d8_isrForCrosstalkSources_1125903_45.stderr parsl.dataflow.dflow INFO: Task 3542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cb50d799b62ecd9ca74418ed8665489 with result from task 3542 parsl.dataflow.dflow DEBUG: Task 3542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/351a2108-a1ca-42ce-bba9-b936531c47b8_isrForCrosstalkSources_1125902_9.stdout parsl.dataflow.dflow INFO: Standard error for task 3542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/351a2108-a1ca-42ce-bba9-b936531c47b8_isrForCrosstalkSources_1125902_9.stderr parsl.dataflow.dflow INFO: Task 3654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca6339259e86167865c5479c42b593c1 with result from task 3654 parsl.dataflow.dflow DEBUG: Task 3654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f481f751-3ddb-4f6c-96d4-7d108a9803c1_isrForCrosstalkSources_1125939_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125939/f481f751-3ddb-4f6c-96d4-7d108a9803c1_isrForCrosstalkSources_1125939_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:46 proc_lsst.multi:146 [INFO] found job 16896584 in provider checkpoint proc_lsst.multi INFO: found job 16896584 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de7b066e1d0728515387a83a7b15b603 with result from task 3546 parsl.dataflow.dflow DEBUG: Task 3546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/8a6b8317-cc89-445a-b419-9555e9b6db4a_isrForCrosstalkSources_1125909_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/8a6b8317-cc89-445a-b419-9555e9b6db4a_isrForCrosstalkSources_1125909_17.stderr parsl.dataflow.dflow INFO: Task 3617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd135e512771bda972e29122cb92c13c with result from task 3617 parsl.dataflow.dflow DEBUG: Task 3617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/71e69eaa-c24f-4a18-831f-c02967950e0d_isrForCrosstalkSources_1125890_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/71e69eaa-c24f-4a18-831f-c02967950e0d_isrForCrosstalkSources_1125890_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:46 proc_lsst.multi:146 [INFO] found job 16896585 in provider checkpoint proc_lsst.multi INFO: found job 16896585 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21d1dd613c0962b56c987f1357d2c152 with result from task 3591 parsl.dataflow.dflow DEBUG: Task 3591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/86e9b3db-e586-4111-b87d-ff75d563960e_isrForCrosstalkSources_1125937_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/86e9b3db-e586-4111-b87d-ff75d563960e_isrForCrosstalkSources_1125937_34.stderr parsl.dataflow.dflow INFO: Task 3695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03a8d6e260607c98b90e3b6ec20a05ed with result from task 3695 parsl.dataflow.dflow DEBUG: Task 3695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/81b039f1-6f17-404f-b999-ed031dd7fd2d_isrForCrosstalkSources_1125889_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/81b039f1-6f17-404f-b999-ed031dd7fd2d_isrForCrosstalkSources_1125889_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3624 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 34727018fa9f70218f0c3a1de184e450 with result from task 3624 parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3624 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/b0b80693-8b64-4f70-8d68-5c9965de6359_isrForCrosstalkSources_1125935_27.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/b0b80693-8b64-4f70-8d68-5c9965de6359_isrForCrosstalkSources_1125935_27.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:47 proc_lsst.multi:146 [INFO] found job 16896586 in provider checkpoint proc_lsst.multi INFO: found job 16896586 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd6e4a0fc567ad98dff471fe345ed0b9 with result from task 3596 parsl.dataflow.dflow DEBUG: Task 3596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/70c4a5e5-9caf-4c91-a438-73385057da1a_isrForCrosstalkSources_1125928_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/70c4a5e5-9caf-4c91-a438-73385057da1a_isrForCrosstalkSources_1125928_33.stderr parsl.dataflow.dflow INFO: Task 3670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e7a291677616b715060b8b7d9e9cc81 with result from task 3670 parsl.dataflow.dflow DEBUG: Task 3670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/e2bc822f-29c2-4cad-85bd-41d63fd0f9f0_isrForCrosstalkSources_1125900_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/e2bc822f-29c2-4cad-85bd-41d63fd0f9f0_isrForCrosstalkSources_1125900_2.stderr parsl.dataflow.dflow INFO: Task 3584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebd82663ec915f65f9d1fd58c1af4630 with result from task 3584 parsl.dataflow.dflow DEBUG: Task 3584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/164897fb-ac12-4d77-9193-945d0d9ff2e7_isrForCrosstalkSources_1125937_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/164897fb-ac12-4d77-9193-945d0d9ff2e7_isrForCrosstalkSources_1125937_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:47 proc_lsst.multi:146 [INFO] found job 16896587 in provider checkpoint proc_lsst.multi INFO: found job 16896587 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 542d61080db8b30f259294049a77e9ba with result from task 3606 parsl.dataflow.dflow DEBUG: Task 3606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/73c8f704-fdbd-4ec9-974c-abdb1c58286d_isrForCrosstalkSources_1125902_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/73c8f704-fdbd-4ec9-974c-abdb1c58286d_isrForCrosstalkSources_1125902_47.stderr parsl.dataflow.dflow INFO: Task 3578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d4721555b74b8192849d4e5f91063e3 with result from task 3578 parsl.dataflow.dflow DEBUG: Task 3578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/38f7f124-6c42-423a-ac59-e825e7ce6cb7_isrForCrosstalkSources_1125898_34.stdout parsl.dataflow.dflow INFO: Standard error for task 3578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/38f7f124-6c42-423a-ac59-e825e7ce6cb7_isrForCrosstalkSources_1125898_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3642 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 01c6bede693ef8eb8054673db08eb85d with result from task 3642 parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3642 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/ce521ac3-1c57-4f4e-8b9d-01f1f059648b_isrForCrosstalkSources_1125907_27.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125907/ce521ac3-1c57-4f4e-8b9d-01f1f059648b_isrForCrosstalkSources_1125907_27.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:47 proc_lsst.multi:146 [INFO] found job 16896588 in provider checkpoint proc_lsst.multi INFO: found job 16896588 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6595ce397d5fd63af2b943f1c2a145c6 with result from task 3717 parsl.dataflow.dflow DEBUG: Task 3717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/be3a84f5-a8f8-4dba-ab8e-effe19f171f8_isrForCrosstalkSources_1125891_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/be3a84f5-a8f8-4dba-ab8e-effe19f171f8_isrForCrosstalkSources_1125891_18.stderr parsl.dataflow.dflow INFO: Task 3631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 430475b10e1d9796b534316af46733c7 with result from task 3631 parsl.dataflow.dflow DEBUG: Task 3631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/03f2d816-872d-4def-9476-bf740695ef9b_isrForCrosstalkSources_1125909_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/03f2d816-872d-4def-9476-bf740695ef9b_isrForCrosstalkSources_1125909_43.stderr parsl.dataflow.dflow INFO: Task 3644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a42c592de3c47a4ce2d60c29ee02fd5 with result from task 3644 parsl.dataflow.dflow DEBUG: Task 3644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/01eab606-2192-4c72-894d-b8aa67f41ba7_isrForCrosstalkSources_1125917_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/01eab606-2192-4c72-894d-b8aa67f41ba7_isrForCrosstalkSources_1125917_37.stderr parsl.dataflow.dflow INFO: Task 3625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 781777ccfa8bed865a80f14b1330003f with result from task 3625 parsl.dataflow.dflow DEBUG: Task 3625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/1e787b0b-d6a5-434a-a911-810bad0e64ad_isrForCrosstalkSources_1125891_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/1e787b0b-d6a5-434a-a911-810bad0e64ad_isrForCrosstalkSources_1125891_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:47 proc_lsst.multi:146 [INFO] found job 16896589 in provider checkpoint proc_lsst.multi INFO: found job 16896589 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dea231b50edc83ae4d991b1d01eb1eec with result from task 3598 parsl.dataflow.dflow DEBUG: Task 3598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a5b3f2ff-663e-4a9d-89c0-f68a9cf929ae_isrForCrosstalkSources_1125945_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125945/a5b3f2ff-663e-4a9d-89c0-f68a9cf929ae_isrForCrosstalkSources_1125945_48.stderr parsl.dataflow.dflow INFO: Task 3609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f43ed4c4bd2c882d17366a32b5fe1c0f with result from task 3609 parsl.dataflow.dflow DEBUG: Task 3609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/4545329a-05bb-4fc9-aa51-a7e9ee2b607c_isrForCrosstalkSources_1125914_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/4545329a-05bb-4fc9-aa51-a7e9ee2b607c_isrForCrosstalkSources_1125914_49.stderr parsl.dataflow.dflow INFO: Task 3702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a89adc5f0d5c700be2b9c4f4e4e2016 with result from task 3702 parsl.dataflow.dflow DEBUG: Task 3702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/0c14f014-5abe-4adb-9230-6941dbec48f7_isrForCrosstalkSources_1125923_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125923/0c14f014-5abe-4adb-9230-6941dbec48f7_isrForCrosstalkSources_1125923_18.stderr parsl.dataflow.dflow INFO: Task 3608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05da69b4f5d95b189deb3b35b47408d0 with result from task 3608 parsl.dataflow.dflow DEBUG: Task 3608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/2d6aa942-d853-4111-ad9d-55fe1203b641_isrForCrosstalkSources_1125946_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/2d6aa942-d853-4111-ad9d-55fe1203b641_isrForCrosstalkSources_1125946_36.stderr parsl.dataflow.dflow INFO: Task 3563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa9d83e9c120c61f66f30959e611a2a1 with result from task 3563 parsl.dataflow.dflow DEBUG: Task 3563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/2659ae4c-adc9-4161-9f24-a3391e511350_isrForCrosstalkSources_1125940_37.stdout parsl.dataflow.dflow INFO: Standard error for task 3563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/2659ae4c-adc9-4161-9f24-a3391e511350_isrForCrosstalkSources_1125940_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3685 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry a0d070e1a25147d1cbb220388f996015 with result from task 3685 parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3685 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/38853f60-f253-4c61-9361-1e418814bc99_isrForCrosstalkSources_1125920_27.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/38853f60-f253-4c61-9361-1e418814bc99_isrForCrosstalkSources_1125920_27.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3668 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry d440823dfdfef34d9b18db1d4edb6ef0 with result from task 3668 parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3668 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/3b05418f-cb9c-4e88-a2e5-3a016edcab36_isrForCrosstalkSources_1125919_31.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/3b05418f-cb9c-4e88-a2e5-3a016edcab36_isrForCrosstalkSources_1125919_31.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:48 proc_lsst.multi:146 [INFO] found job 16896590 in provider checkpoint proc_lsst.multi INFO: found job 16896590 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69662969415c9fd6a226d2e20b3b4b2d with result from task 3712 parsl.dataflow.dflow DEBUG: Task 3712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/76cd4f4e-ac34-4988-9eb7-eae319839394_isrForCrosstalkSources_1125902_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/76cd4f4e-ac34-4988-9eb7-eae319839394_isrForCrosstalkSources_1125902_7.stderr parsl.dataflow.dflow INFO: Task 3637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffeea7c3f64ec5860d2688432db6a8ff with result from task 3637 parsl.dataflow.dflow DEBUG: Task 3637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/898914bd-234d-4c0b-a689-74bc26209ea7_isrForCrosstalkSources_1125888_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125888/898914bd-234d-4c0b-a689-74bc26209ea7_isrForCrosstalkSources_1125888_47.stderr parsl.dataflow.dflow INFO: Task 3667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d607c2428685a3ed61c379ab81a56ab7 with result from task 3667 parsl.dataflow.dflow DEBUG: Task 3667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/3ea3e9b3-1d55-49c5-80c2-650659a7914b_isrForCrosstalkSources_1125898_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/3ea3e9b3-1d55-49c5-80c2-650659a7914b_isrForCrosstalkSources_1125898_24.stderr parsl.dataflow.dflow INFO: Task 3612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a6c9e7443c36683a375725a0d7f32a3 with result from task 3612 parsl.dataflow.dflow DEBUG: Task 3612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/b2a6d867-f22d-4e12-94c7-e91da4054e1e_isrForCrosstalkSources_1125911_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/b2a6d867-f22d-4e12-94c7-e91da4054e1e_isrForCrosstalkSources_1125911_41.stderr parsl.dataflow.dflow INFO: Task 3706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 176a3af87e09ec2c515b3a5b63b3039e with result from task 3706 parsl.dataflow.dflow DEBUG: Task 3706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/15cbd580-ce6e-43be-aa64-ccb7c343e1e6_isrForCrosstalkSources_1125935_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/15cbd580-ce6e-43be-aa64-ccb7c343e1e6_isrForCrosstalkSources_1125935_36.stderr parsl.dataflow.dflow INFO: Task 3700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 588be9808173f099005f8ace351397b4 with result from task 3700 parsl.dataflow.dflow DEBUG: Task 3700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/94f8d5d2-50f9-4f03-81c3-33cc1f122792_isrForCrosstalkSources_1125929_42.stdout parsl.dataflow.dflow INFO: Standard error for task 3700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/94f8d5d2-50f9-4f03-81c3-33cc1f122792_isrForCrosstalkSources_1125929_42.stderr parsl.dataflow.dflow INFO: Task 3682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 998c0442da34d924abb6bc3e5336bf60 with result from task 3682 parsl.dataflow.dflow DEBUG: Task 3682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/c1272209-7fd4-48b2-a275-4c55a1e039ef_isrForCrosstalkSources_1125930_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/c1272209-7fd4-48b2-a275-4c55a1e039ef_isrForCrosstalkSources_1125930_2.stderr parsl.dataflow.dflow INFO: Task 3573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcb6d809571b495add26815978c2542e with result from task 3573 parsl.dataflow.dflow DEBUG: Task 3573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/d9c09e9d-4275-4018-a0cf-503a67903d90_isrForCrosstalkSources_1125906_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/d9c09e9d-4275-4018-a0cf-503a67903d90_isrForCrosstalkSources_1125906_3.stderr parsl.dataflow.dflow INFO: Task 3650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32999bec2068d450c48ddc590e3898d3 with result from task 3650 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:48 proc_lsst.multi:146 [INFO] found job 16896591 in provider checkpoint proc_lsst.multi INFO: found job 16896591 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow DEBUG: Task 3650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/dfd97ceb-f4d0-4716-9e32-e73a9590666c_isrForCrosstalkSources_1125919_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/dfd97ceb-f4d0-4716-9e32-e73a9590666c_isrForCrosstalkSources_1125919_3.stderr parsl.dataflow.dflow INFO: Task 3593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b32b439e5c8dfc55f0b76f95b308304 with result from task 3593 parsl.dataflow.dflow DEBUG: Task 3593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/bce4c1b2-c47c-4513-9d3e-45292ff415f3_isrForCrosstalkSources_1125894_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/bce4c1b2-c47c-4513-9d3e-45292ff415f3_isrForCrosstalkSources_1125894_47.stderr parsl.dataflow.dflow INFO: Task 3641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6aa7d1f0977d22cac47fa9b1472005fe with result from task 3641 parsl.dataflow.dflow DEBUG: Task 3641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/877d790f-0ef5-43a5-9c42-e6ddbf5681e2_isrForCrosstalkSources_1125919_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/877d790f-0ef5-43a5-9c42-e6ddbf5681e2_isrForCrosstalkSources_1125919_11.stderr parsl.dataflow.dflow INFO: Task 3718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66848c9383fc38615bd41294865d4388 with result from task 3718 parsl.dataflow.dflow DEBUG: Task 3718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/5d47d6df-d318-43d2-aaad-64a3cbb96151_isrForCrosstalkSources_1125931_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125931/5d47d6df-d318-43d2-aaad-64a3cbb96151_isrForCrosstalkSources_1125931_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:48 proc_lsst.multi:146 [INFO] found job 16896592 in provider checkpoint proc_lsst.multi INFO: found job 16896592 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d40af925dbf3dcf8c5f94da2fe7b6a53 with result from task 3622 parsl.dataflow.dflow DEBUG: Task 3622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/1e5f09a2-a4cd-485c-9faa-8e614248fc14_isrForCrosstalkSources_1125898_22.stdout parsl.dataflow.dflow INFO: Standard error for task 3622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/1e5f09a2-a4cd-485c-9faa-8e614248fc14_isrForCrosstalkSources_1125898_22.stderr parsl.dataflow.dflow INFO: Task 3698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 479849bf1e67eb88999d7b35e942b9fa with result from task 3698 parsl.dataflow.dflow DEBUG: Task 3698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/2b2ccc1f-fb50-4c4d-a711-42a6be8f1d57_isrForCrosstalkSources_1125901_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125901/2b2ccc1f-fb50-4c4d-a711-42a6be8f1d57_isrForCrosstalkSources_1125901_4.stderr parsl.dataflow.dflow INFO: Task 3511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00c3ea9edb89c493f61d61fee62af2a0 with result from task 3511 parsl.dataflow.dflow DEBUG: Task 3511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/09a36002-d375-4d28-aed0-42290c7574b8_isrForCrosstalkSources_1125889_47.stdout parsl.dataflow.dflow INFO: Standard error for task 3511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125889/09a36002-d375-4d28-aed0-42290c7574b8_isrForCrosstalkSources_1125889_47.stderr parsl.dataflow.dflow INFO: Task 3663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60337c47c166f224e27fb8f01c247233 with result from task 3663 parsl.dataflow.dflow DEBUG: Task 3663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d60a0674-b9c1-4647-a7e1-d2db9dae0d0b_isrForCrosstalkSources_1125919_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125919/d60a0674-b9c1-4647-a7e1-d2db9dae0d0b_isrForCrosstalkSources_1125919_26.stderr parsl.dataflow.dflow INFO: Task 3681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87e441ab318c1281d629c45c0ed4d9b5 with result from task 3681 parsl.dataflow.dflow DEBUG: Task 3681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/7b27a9b4-f79c-40bb-a5c8-6cbb59a8d9b8_isrForCrosstalkSources_1125890_49.stdout parsl.dataflow.dflow INFO: Standard error for task 3681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/7b27a9b4-f79c-40bb-a5c8-6cbb59a8d9b8_isrForCrosstalkSources_1125890_49.stderr parsl.dataflow.dflow INFO: Task 3673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 462f92662b19457f3bc40f0e1178ffc1 with result from task 3673 parsl.dataflow.dflow DEBUG: Task 3673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/1d695cf5-5b72-48e6-8c9e-7bfd7246b85f_isrForCrosstalkSources_1125944_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125944/1d695cf5-5b72-48e6-8c9e-7bfd7246b85f_isrForCrosstalkSources_1125944_6.stderr parsl.dataflow.dflow INFO: Task 3621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aae237690e049ff9d6a10a2e8d67de20 with result from task 3621 parsl.dataflow.dflow DEBUG: Task 3621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/baef05cc-566f-4907-ad0b-afa265827692_isrForCrosstalkSources_1125925_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/baef05cc-566f-4907-ad0b-afa265827692_isrForCrosstalkSources_1125925_44.stderr parsl.dataflow.dflow INFO: Task 3689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa913ff793984bd306379da90d7cec4a with result from task 3689 parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3689 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ec1dbc84-cb33-4666-a89d-87553eabcc4e_isrForCrosstalkSources_1125902_50.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/ec1dbc84-cb33-4666-a89d-87553eabcc4e_isrForCrosstalkSources_1125902_50.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3550 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry e0409fd5615118bdcb598bd6252cc004 with result from task 3550 parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3550 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/3647c19f-7ee6-492e-bf7e-dd7a7c4df4c6_isrForCrosstalkSources_1125904_3.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/3647c19f-7ee6-492e-bf7e-dd7a7c4df4c6_isrForCrosstalkSources_1125904_3.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:48 proc_lsst.multi:146 [INFO] found job 16896593 in provider checkpoint proc_lsst.multi INFO: found job 16896593 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d889c6d1c6ff0823cb824ab5acc55634 with result from task 3636 parsl.dataflow.dflow DEBUG: Task 3636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/0381c06e-45f0-49f3-a593-78993afdef87_isrForCrosstalkSources_1125909_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/0381c06e-45f0-49f3-a593-78993afdef87_isrForCrosstalkSources_1125909_48.stderr parsl.dataflow.dflow INFO: Task 3680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry caafc25ba7f396d7728dbcec055bffbe with result from task 3680 parsl.dataflow.dflow DEBUG: Task 3680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/f52cf7cc-984c-4732-b902-e8a1496685dd_isrForCrosstalkSources_1125937_4.stdout parsl.dataflow.dflow INFO: Standard error for task 3680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/f52cf7cc-984c-4732-b902-e8a1496685dd_isrForCrosstalkSources_1125937_4.stderr parsl.dataflow.dflow INFO: Task 3629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fc8ebbb532711267a1f274104c66ff2 with result from task 3629 parsl.dataflow.dflow DEBUG: Task 3629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/a6eedaeb-46a6-48a4-a07d-a80a6076f9c6_isrForCrosstalkSources_1125915_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/a6eedaeb-46a6-48a4-a07d-a80a6076f9c6_isrForCrosstalkSources_1125915_38.stderr parsl.dataflow.dflow INFO: Task 3665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c418dd1ddfd6e85f01ad3d7fa897265e with result from task 3665 parsl.dataflow.dflow DEBUG: Task 3665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/e95e886d-3a16-429a-b705-9b93ec2b0a71_isrForCrosstalkSources_1125902_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/e95e886d-3a16-429a-b705-9b93ec2b0a71_isrForCrosstalkSources_1125902_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:48 proc_lsst.multi:146 [INFO] found job 16896594 in provider checkpoint proc_lsst.multi INFO: found job 16896594 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ec5631eea4be31e4a7aa01e7425bee1 with result from task 3627 parsl.dataflow.dflow DEBUG: Task 3627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/4dd67bfa-9c09-47ea-b5e1-c1e471a61fa8_isrForCrosstalkSources_1125913_60.stdout parsl.dataflow.dflow INFO: Standard error for task 3627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125913/4dd67bfa-9c09-47ea-b5e1-c1e471a61fa8_isrForCrosstalkSources_1125913_60.stderr parsl.dataflow.dflow INFO: Task 3677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0519f5a71c09fd499e29f63f84a10447 with result from task 3677 parsl.dataflow.dflow DEBUG: Task 3677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/87d1c460-ac8d-4799-add6-dc43143202c0_isrForCrosstalkSources_1125915_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/87d1c460-ac8d-4799-add6-dc43143202c0_isrForCrosstalkSources_1125915_43.stderr parsl.dataflow.dflow INFO: Task 3686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dce77e4b642e768826c0445ed6837b37 with result from task 3686 parsl.dataflow.dflow DEBUG: Task 3686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/76295ad8-393c-4706-b25c-b40439780766_isrForCrosstalkSources_1125909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125909/76295ad8-393c-4706-b25c-b40439780766_isrForCrosstalkSources_1125909_2.stderr parsl.dataflow.dflow INFO: Task 3551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef9f871bf3e9b610b3926896614d0a4d with result from task 3551 parsl.dataflow.dflow DEBUG: Task 3551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/07a943b8-9b21-441d-a392-c604f4ac1d5c_isrForCrosstalkSources_1125940_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/07a943b8-9b21-441d-a392-c604f4ac1d5c_isrForCrosstalkSources_1125940_55.stderr parsl.dataflow.dflow INFO: Task 3699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30b2019f00c0171d2d1471f0944fda64 with result from task 3699 parsl.dataflow.dflow DEBUG: Task 3699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/75d13548-9d57-4bdf-845c-66dd323d00ba_isrForCrosstalkSources_1125906_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125906/75d13548-9d57-4bdf-845c-66dd323d00ba_isrForCrosstalkSources_1125906_41.stderr parsl.dataflow.dflow INFO: Task 3683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbe7a7b9222c45dabb642ee9ad0c1e92 with result from task 3683 parsl.dataflow.dflow DEBUG: Task 3683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/c4b1a594-1b8c-465b-9eef-62a7b033aca5_isrForCrosstalkSources_1125915_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/c4b1a594-1b8c-465b-9eef-62a7b033aca5_isrForCrosstalkSources_1125915_7.stderr parsl.dataflow.dflow INFO: Task 3568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67388f01a2e95cfe0b143ff035bfd474 with result from task 3568 parsl.dataflow.dflow DEBUG: Task 3568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/3024035f-bac4-413e-864c-d7c05468b472_isrForCrosstalkSources_1125891_17.stdout parsl.dataflow.dflow INFO: Standard error for task 3568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/3024035f-bac4-413e-864c-d7c05468b472_isrForCrosstalkSources_1125891_17.stderr parsl.dataflow.dflow INFO: Task 3697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0488179b695d1311150b31fd74c79efa with result from task 3697 parsl.dataflow.dflow DEBUG: Task 3697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/8b2654fb-3837-4e54-a9a0-8479db6d0d05_isrForCrosstalkSources_1125929_35.stdout parsl.dataflow.dflow INFO: Standard error for task 3697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/8b2654fb-3837-4e54-a9a0-8479db6d0d05_isrForCrosstalkSources_1125929_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:49 proc_lsst.multi:146 [INFO] found job 16896595 in provider checkpoint proc_lsst.multi INFO: found job 16896595 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b1e1057310dd2ff36b2a7cd6d14751f with result from task 3675 parsl.dataflow.dflow DEBUG: Task 3675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/249c871d-0a00-417a-a66c-584724a753c7_isrForCrosstalkSources_1125904_24.stdout parsl.dataflow.dflow INFO: Standard error for task 3675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125904/249c871d-0a00-417a-a66c-584724a753c7_isrForCrosstalkSources_1125904_24.stderr parsl.dataflow.dflow INFO: Task 3632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b499e31c30080d8c831b17a84c6df87 with result from task 3632 parsl.dataflow.dflow DEBUG: Task 3632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a0f0f3c3-5308-4747-838d-f7af5a9064b5_isrForCrosstalkSources_1125900_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/a0f0f3c3-5308-4747-838d-f7af5a9064b5_isrForCrosstalkSources_1125900_38.stderr parsl.dataflow.dflow INFO: Task 3679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30d6d54a692ea5401b7cf60d0be78fcc with result from task 3679 parsl.dataflow.dflow DEBUG: Task 3679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/bcfa0bef-c4f3-4703-8b4e-4ab4e9d42766_isrForCrosstalkSources_1125918_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/bcfa0bef-c4f3-4703-8b4e-4ab4e9d42766_isrForCrosstalkSources_1125918_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3672 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 4418a3bd101d29b52bb7208456ff9d0b with result from task 3672 parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3672 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/df9330ee-6d6e-4059-a69a-dfe5b0aa9c94_isrForCrosstalkSources_1125912_43.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125912/df9330ee-6d6e-4059-a69a-dfe5b0aa9c94_isrForCrosstalkSources_1125912_43.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:49 proc_lsst.multi:146 [INFO] found job 16896596 in provider checkpoint proc_lsst.multi INFO: found job 16896596 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c541f19c4dab15c31d16d8eccb1a92ff with result from task 3691 parsl.dataflow.dflow DEBUG: Task 3691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d40cc16d-373e-4412-be5b-7607739f3ef9_isrForCrosstalkSources_1125915_50.stdout parsl.dataflow.dflow INFO: Standard error for task 3691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d40cc16d-373e-4412-be5b-7607739f3ef9_isrForCrosstalkSources_1125915_50.stderr parsl.dataflow.dflow INFO: Task 3705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1258c336f28dfc35b535f5e50972771d with result from task 3705 parsl.dataflow.dflow DEBUG: Task 3705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/d590dcfa-68e7-46e6-a16e-d1cbfcb00873_isrForCrosstalkSources_1125928_7.stdout parsl.dataflow.dflow INFO: Standard error for task 3705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125928/d590dcfa-68e7-46e6-a16e-d1cbfcb00873_isrForCrosstalkSources_1125928_7.stderr parsl.dataflow.dflow INFO: Task 3666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4684ee1510466517ea860fd7f3e5ecf with result from task 3666 parsl.dataflow.dflow DEBUG: Task 3666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/b3753702-b34e-4927-9cc4-62a154d2961a_isrForCrosstalkSources_1125890_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/b3753702-b34e-4927-9cc4-62a154d2961a_isrForCrosstalkSources_1125890_57.stderr parsl.dataflow.dflow INFO: Task 3662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af10a41270d6fff8ba5fb819053a2783 with result from task 3662 parsl.dataflow.dflow DEBUG: Task 3662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/6927c681-ea3b-4eef-ab39-a1061181c7e1_isrForCrosstalkSources_1125892_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/6927c681-ea3b-4eef-ab39-a1061181c7e1_isrForCrosstalkSources_1125892_5.stderr parsl.dataflow.dflow INFO: Task 3669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 435a263d7ec723590a27b7c55d20547e with result from task 3669 parsl.dataflow.dflow DEBUG: Task 3669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/74b7d879-dd83-4da3-b609-4a055cc2b250_isrForCrosstalkSources_1125915_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/74b7d879-dd83-4da3-b609-4a055cc2b250_isrForCrosstalkSources_1125915_59.stderr parsl.dataflow.dflow INFO: Task 3660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1da32b89648890cabe47b3465f73b80 with result from task 3660 parsl.dataflow.dflow DEBUG: Task 3660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/dcfd9a02-7dc7-4f52-8f11-4b728faac4a6_isrForCrosstalkSources_1125941_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/dcfd9a02-7dc7-4f52-8f11-4b728faac4a6_isrForCrosstalkSources_1125941_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3594 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry bb0b9a6ad0adc4a23e2cf73ac18066c8 with result from task 3594 parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3594 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/14d62464-14cd-43b2-b780-d331cf794127_isrForCrosstalkSources_1125900_29.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125900/14d62464-14cd-43b2-b780-d331cf794127_isrForCrosstalkSources_1125900_29.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:49 proc_lsst.multi:146 [INFO] found job 16896597 in provider checkpoint proc_lsst.multi INFO: found job 16896597 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eaaf3c2899f1d4d7b028c325b22a5e02 with result from task 3653 parsl.dataflow.dflow DEBUG: Task 3653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/ad3383e1-f648-44bc-b601-0e0080853950_isrForCrosstalkSources_1125930_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125930/ad3383e1-f648-44bc-b601-0e0080853950_isrForCrosstalkSources_1125930_15.stderr parsl.dataflow.dflow INFO: Task 3707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86b54064ecc4242afff74b3461ad2de2 with result from task 3707 parsl.dataflow.dflow DEBUG: Task 3707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/b3fa840b-5926-4b81-9cb8-cfa197cd43ab_isrForCrosstalkSources_1125922_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/b3fa840b-5926-4b81-9cb8-cfa197cd43ab_isrForCrosstalkSources_1125922_1.stderr parsl.dataflow.dflow INFO: Task 3701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bcbf71a73020e7594c79c46c4297fdf with result from task 3701 parsl.dataflow.dflow DEBUG: Task 3701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/7de65aa6-1091-4f31-9728-e224424ca62a_isrForCrosstalkSources_1125905_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125905/7de65aa6-1091-4f31-9728-e224424ca62a_isrForCrosstalkSources_1125905_28.stderr parsl.dataflow.dflow INFO: Task 3651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5aad89c8a3e84d68f04a8c5f9817047b with result from task 3651 parsl.dataflow.dflow DEBUG: Task 3651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/513c862f-759f-4a3d-a751-3b0139c92d58_isrForCrosstalkSources_1125935_38.stdout parsl.dataflow.dflow INFO: Standard error for task 3651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/513c862f-759f-4a3d-a751-3b0139c92d58_isrForCrosstalkSources_1125935_38.stderr parsl.dataflow.dflow INFO: Task 3694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa83539beedb56d0a003c8d7cc35befb with result from task 3694 parsl.dataflow.dflow DEBUG: Task 3694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/672288d2-cab9-4349-81ec-633febf46563_isrForCrosstalkSources_1125946_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125946/672288d2-cab9-4349-81ec-633febf46563_isrForCrosstalkSources_1125946_59.stderr parsl.dataflow.dflow INFO: Task 3655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df64a31d3012a38df2336b393907d180 with result from task 3655 parsl.dataflow.dflow DEBUG: Task 3655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/18a16579-2980-4923-a15b-8fe24685d643_isrForCrosstalkSources_1125932_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/18a16579-2980-4923-a15b-8fe24685d643_isrForCrosstalkSources_1125932_39.stderr parsl.dataflow.dflow INFO: Task 3690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8d0118147bb41063b91e40c878f753c with result from task 3690 parsl.dataflow.dflow DEBUG: Task 3690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/9e03135c-1711-4750-a36b-2e375d1e6514_isrForCrosstalkSources_1125940_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/9e03135c-1711-4750-a36b-2e375d1e6514_isrForCrosstalkSources_1125940_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:49 proc_lsst.multi:146 [INFO] found job 16896598 in provider checkpoint proc_lsst.multi INFO: found job 16896598 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c9469108f1d5701ced0b620ab2c54a0 with result from task 3646 parsl.dataflow.dflow DEBUG: Task 3646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/91c76eeb-bc7f-429f-9b42-d982f9120a26_isrForCrosstalkSources_1125890_10.stdout parsl.dataflow.dflow INFO: Standard error for task 3646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125890/91c76eeb-bc7f-429f-9b42-d982f9120a26_isrForCrosstalkSources_1125890_10.stderr parsl.dataflow.dflow INFO: Task 3704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7df2c7e5926fa37be874a4118e730692 with result from task 3704 parsl.dataflow.dflow DEBUG: Task 3704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/375729e2-c542-41c5-bd64-04361b54919f_isrForCrosstalkSources_1125940_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125940/375729e2-c542-41c5-bd64-04361b54919f_isrForCrosstalkSources_1125940_23.stderr parsl.dataflow.dflow INFO: Task 3658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8704145e09936a9280bfc8a919ab825e with result from task 3658 parsl.dataflow.dflow DEBUG: Task 3658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/50f96941-64e9-411d-bc5c-f2a7683b566a_isrForCrosstalkSources_1125942_33.stdout parsl.dataflow.dflow INFO: Standard error for task 3658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125942/50f96941-64e9-411d-bc5c-f2a7683b566a_isrForCrosstalkSources_1125942_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:49 proc_lsst.multi:146 [INFO] found job 16896599 in provider checkpoint proc_lsst.multi INFO: found job 16896599 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:50 proc_lsst.multi:146 [INFO] found job 16896600 in provider checkpoint proc_lsst.multi INFO: found job 16896600 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c52dccfdde903d54727d17f7655308f6 with result from task 3684 parsl.dataflow.dflow DEBUG: Task 3684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/be710b1c-4524-4518-b36c-5bc09c7376c1_isrForCrosstalkSources_1125922_41.stdout parsl.dataflow.dflow INFO: Standard error for task 3684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/be710b1c-4524-4518-b36c-5bc09c7376c1_isrForCrosstalkSources_1125922_41.stderr parsl.dataflow.dflow INFO: Task 3674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd8aabfe6dc64a4e103bae6dd87144d9 with result from task 3674 parsl.dataflow.dflow DEBUG: Task 3674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f2542e42-21bd-4b97-848a-6e7f2704e2ad_isrForCrosstalkSources_1125897_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125897/f2542e42-21bd-4b97-848a-6e7f2704e2ad_isrForCrosstalkSources_1125897_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:50 proc_lsst.multi:146 [INFO] found job 16896601 in provider checkpoint proc_lsst.multi INFO: found job 16896601 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f8ae665caf9ce2b504dc88df466bb2c with result from task 3676 parsl.dataflow.dflow DEBUG: Task 3676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/872b128a-abca-47c2-80f1-4a4221d4c176_isrForCrosstalkSources_1125898_36.stdout parsl.dataflow.dflow INFO: Standard error for task 3676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125898/872b128a-abca-47c2-80f1-4a4221d4c176_isrForCrosstalkSources_1125898_36.stderr parsl.dataflow.dflow INFO: Task 3708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8de949b43ac16a8dfb722ebc4a624f74 with result from task 3708 parsl.dataflow.dflow DEBUG: Task 3708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/90895def-532b-4031-a2df-f65562341e35_isrForCrosstalkSources_1125922_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/90895def-532b-4031-a2df-f65562341e35_isrForCrosstalkSources_1125922_21.stderr parsl.dataflow.dflow INFO: Task 3692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2043040cd4dad92999215b43f631666e with result from task 3692 parsl.dataflow.dflow DEBUG: Task 3692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/1c479cd6-7049-4db3-9ceb-d3706a2bb223_isrForCrosstalkSources_1125935_12.stdout parsl.dataflow.dflow INFO: Standard error for task 3692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125935/1c479cd6-7049-4db3-9ceb-d3706a2bb223_isrForCrosstalkSources_1125935_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:50 proc_lsst.multi:146 [INFO] found job 16896602 in provider checkpoint proc_lsst.multi INFO: found job 16896602 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1f5a5f104828dad49eaa60b0ed15b4b with result from task 3661 parsl.dataflow.dflow DEBUG: Task 3661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/ac99be26-18ac-4032-82d6-46f95a22fdb8_isrForCrosstalkSources_1125936_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125936/ac99be26-18ac-4032-82d6-46f95a22fdb8_isrForCrosstalkSources_1125936_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Task 3643 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.memoization DEBUG: Storing app cache entry 22ab1ca65e8f7ea90d0bc9a9717b02df with result from task 3643 parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow DEBUG: Task 3643 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard output for task 3643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d10386d0-d768-47cc-95e5-a5fe0df6609e_isrForCrosstalkSources_1125915_5.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.dataflow.dflow INFO: Standard error for task 3643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125915/d10386d0-d768-47cc-95e5-a5fe0df6609e_isrForCrosstalkSources_1125915_5.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:50 proc_lsst.multi:146 [INFO] found job 16896603 in provider checkpoint proc_lsst.multi INFO: found job 16896603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47b9b8dfbfa36dd72b946be434c98b5b with result from task 3652 parsl.dataflow.dflow DEBUG: Task 3652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/7fbffb57-f32c-429e-9960-d35102216a46_isrForCrosstalkSources_1125920_29.stdout parsl.dataflow.dflow INFO: Standard error for task 3652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125920/7fbffb57-f32c-429e-9960-d35102216a46_isrForCrosstalkSources_1125920_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:50 proc_lsst.multi:146 [INFO] found job 16896604 in provider checkpoint proc_lsst.multi INFO: found job 16896604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 119bab941c550dd995f8eae57d30cc6d with result from task 3618 parsl.dataflow.dflow DEBUG: Task 3618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/09dfc302-c179-4c0f-9c30-b6ffd2335e70_isrForCrosstalkSources_1125937_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125937/09dfc302-c179-4c0f-9c30-b6ffd2335e70_isrForCrosstalkSources_1125937_2.stderr parsl.dataflow.dflow INFO: Task 3649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45f291f234485d7f6808cd25baced51f with result from task 3649 parsl.dataflow.dflow DEBUG: Task 3649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/42467f8a-3a79-4390-99a9-cd0bce27c2b9_isrForCrosstalkSources_1125933_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125933/42467f8a-3a79-4390-99a9-cd0bce27c2b9_isrForCrosstalkSources_1125933_48.stderr parsl.dataflow.dflow INFO: Task 3709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5088e905601e3dfcf29c6dff1817e8e0 with result from task 3709 parsl.dataflow.dflow DEBUG: Task 3709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/2fe0c35e-202b-47f1-bbd2-a1f1f5f9a04f_isrForCrosstalkSources_1125932_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125932/2fe0c35e-202b-47f1-bbd2-a1f1f5f9a04f_isrForCrosstalkSources_1125932_13.stderr parsl.dataflow.dflow INFO: Task 3574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1706da7b37229b5cd5d2036cdb97441f with result from task 3574 parsl.dataflow.dflow DEBUG: Task 3574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/69b3af32-fce1-4600-a338-96abaa8cae6d_isrForCrosstalkSources_1125922_59.stdout parsl.dataflow.dflow INFO: Standard error for task 3574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/69b3af32-fce1-4600-a338-96abaa8cae6d_isrForCrosstalkSources_1125922_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:51 proc_lsst.multi:146 [INFO] found job 16896605 in provider checkpoint proc_lsst.multi INFO: found job 16896605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:51 proc_lsst.multi:146 [INFO] found job 16896606 in provider checkpoint proc_lsst.multi INFO: found job 16896606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a0b686af38ab9346bddb338abea42a0 with result from task 3645 parsl.dataflow.dflow DEBUG: Task 3645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/2f3bb024-9f8c-4fcf-a4c9-58d40e06e0bc_isrForCrosstalkSources_1125917_11.stdout parsl.dataflow.dflow INFO: Standard error for task 3645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/2f3bb024-9f8c-4fcf-a4c9-58d40e06e0bc_isrForCrosstalkSources_1125917_11.stderr parsl.dataflow.dflow INFO: Task 3719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a317561ffa094a1e2993393108ee1eb with result from task 3719 parsl.dataflow.dflow DEBUG: Task 3719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/ee8fdaa0-0d94-4e3f-83ae-bb19f962adc7_isrForCrosstalkSources_1125891_6.stdout parsl.dataflow.dflow INFO: Standard error for task 3719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125891/ee8fdaa0-0d94-4e3f-83ae-bb19f962adc7_isrForCrosstalkSources_1125891_6.stderr parsl.dataflow.dflow INFO: Task 3687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d86a2b96d3d97e977d004be1aa45b80 with result from task 3687 parsl.dataflow.dflow DEBUG: Task 3687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/bcf5f891-c6e1-4cba-a69b-b2a6c91639b7_isrForCrosstalkSources_1125943_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125943/bcf5f891-c6e1-4cba-a69b-b2a6c91639b7_isrForCrosstalkSources_1125943_19.stderr parsl.dataflow.dflow INFO: Task 3659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17c681e5254d821fd81d60d95432cad6 with result from task 3659 parsl.dataflow.dflow DEBUG: Task 3659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/4bef2bef-e47f-4f7a-b293-51009c896389_isrForCrosstalkSources_1125917_23.stdout parsl.dataflow.dflow INFO: Standard error for task 3659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125917/4bef2bef-e47f-4f7a-b293-51009c896389_isrForCrosstalkSources_1125917_23.stderr parsl.dataflow.dflow INFO: Task 3703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c80365a1891ebfe367d2aecf8b259c4c with result from task 3703 parsl.dataflow.dflow DEBUG: Task 3703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/19430e54-24ef-43e4-a008-150b1be860ad_isrForCrosstalkSources_1125926_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/19430e54-24ef-43e4-a008-150b1be860ad_isrForCrosstalkSources_1125926_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:51 proc_lsst.multi:146 [INFO] found job 16896607 in provider checkpoint proc_lsst.multi INFO: found job 16896607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89a43428e4ff769232c8e3b904abf90a with result from task 3710 parsl.dataflow.dflow DEBUG: Task 3710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/45af3afb-6af1-44c7-9f19-145f5aead9a2_isrForCrosstalkSources_1125925_1.stdout parsl.dataflow.dflow INFO: Standard error for task 3710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125925/45af3afb-6af1-44c7-9f19-145f5aead9a2_isrForCrosstalkSources_1125925_1.stderr parsl.dataflow.dflow INFO: Task 3716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8c64ae36d3ef0db16c33adef7dabba3 with result from task 3716 parsl.dataflow.dflow DEBUG: Task 3716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/d1ea00f0-054a-46bb-aaf2-67aa09b1a9ea_isrForCrosstalkSources_1125926_8.stdout parsl.dataflow.dflow INFO: Standard error for task 3716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125926/d1ea00f0-054a-46bb-aaf2-67aa09b1a9ea_isrForCrosstalkSources_1125926_8.stderr parsl.dataflow.dflow INFO: Task 3711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3be88c84fb20087bcecc084f1d03d49 with result from task 3711 parsl.dataflow.dflow DEBUG: Task 3711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/692452d7-c829-4a29-a636-1262a9baae8e_isrForCrosstalkSources_1125916_25.stdout parsl.dataflow.dflow INFO: Standard error for task 3711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125916/692452d7-c829-4a29-a636-1262a9baae8e_isrForCrosstalkSources_1125916_25.stderr parsl.dataflow.dflow INFO: Task 3647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 808d920009e427cd65725fe5a5060803 with result from task 3647 parsl.dataflow.dflow DEBUG: Task 3647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/51bc3091-db89-4033-b9da-042790a1fd98_isrForCrosstalkSources_1125899_18.stdout parsl.dataflow.dflow INFO: Standard error for task 3647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125899/51bc3091-db89-4033-b9da-042790a1fd98_isrForCrosstalkSources_1125899_18.stderr parsl.dataflow.dflow INFO: Task 3713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 180300531d60ada59f456d91a8818920 with result from task 3713 parsl.dataflow.dflow DEBUG: Task 3713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/36794827-8277-499d-a62f-cc5872d0f2fb_isrForCrosstalkSources_1125911_26.stdout parsl.dataflow.dflow INFO: Standard error for task 3713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125911/36794827-8277-499d-a62f-cc5872d0f2fb_isrForCrosstalkSources_1125911_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:51 proc_lsst.multi:146 [INFO] found job 16896608 in provider checkpoint proc_lsst.multi INFO: found job 16896608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:51 proc_lsst.multi:146 [INFO] found job 16896609 in provider checkpoint proc_lsst.multi INFO: found job 16896609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84d2c336370b87a9dbcfa24641447cfb with result from task 3671 parsl.dataflow.dflow DEBUG: Task 3671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/c15bf00c-0958-44af-a767-76c4f4b059e8_isrForCrosstalkSources_1125929_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125929/c15bf00c-0958-44af-a767-76c4f4b059e8_isrForCrosstalkSources_1125929_30.stderr parsl.dataflow.dflow INFO: Task 3678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e980558ae94bb34468af61300e4d10b0 with result from task 3678 parsl.dataflow.dflow DEBUG: Task 3678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/aadcf80d-4892-4720-ae94-19a508eb6680_isrForCrosstalkSources_1125941_39.stdout parsl.dataflow.dflow INFO: Standard error for task 3678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125941/aadcf80d-4892-4720-ae94-19a508eb6680_isrForCrosstalkSources_1125941_39.stderr parsl.dataflow.dflow INFO: Task 3620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef21e2dbd9b9fd79d772a4da5c76154e with result from task 3620 parsl.dataflow.dflow DEBUG: Task 3620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0eb050a1-942b-4e11-a022-a60bb4294e6d_isrForCrosstalkSources_1125914_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125914/0eb050a1-942b-4e11-a022-a60bb4294e6d_isrForCrosstalkSources_1125914_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:52 proc_lsst.multi:146 [INFO] found job 16896610 in provider checkpoint proc_lsst.multi INFO: found job 16896610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 577182a2a8355a098bb9578e45ce74b7 with result from task 3715 parsl.dataflow.dflow DEBUG: Task 3715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/0731bc6b-98ce-4004-adec-0aec5594ffd2_isrForCrosstalkSources_1125922_5.stdout parsl.dataflow.dflow INFO: Standard error for task 3715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125922/0731bc6b-98ce-4004-adec-0aec5594ffd2_isrForCrosstalkSources_1125922_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:52 proc_lsst.multi:146 [INFO] found job 16896611 in provider checkpoint proc_lsst.multi INFO: found job 16896611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b59401d3c6c9e697e42ea557f179a91e with result from task 3714 parsl.dataflow.dflow DEBUG: Task 3714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/6f19b8b8-2d4b-47e6-b2a1-0ed0beb500a7_isrForCrosstalkSources_1125902_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125902/6f19b8b8-2d4b-47e6-b2a1-0ed0beb500a7_isrForCrosstalkSources_1125902_30.stderr parsl.dataflow.dflow INFO: Task 3656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14e58f85acc27ad22132b8bcb28ae50b with result from task 3656 parsl.dataflow.dflow DEBUG: Task 3656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/c2e919b2-fa2c-4cd8-9164-ab479837b875_isrForCrosstalkSources_1125894_56.stdout parsl.dataflow.dflow INFO: Standard error for task 3656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125894/c2e919b2-fa2c-4cd8-9164-ab479837b875_isrForCrosstalkSources_1125894_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:52 proc_lsst.multi:146 [INFO] found job 16896612 in provider checkpoint proc_lsst.multi INFO: found job 16896612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96c5656af0ce9047dc4feaa3ee95f6b5 with result from task 3688 parsl.dataflow.dflow DEBUG: Task 3688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/f64ae197-7b11-41be-93d4-4556deb2118b_isrForCrosstalkSources_1125947_28.stdout parsl.dataflow.dflow INFO: Standard error for task 3688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125947/f64ae197-7b11-41be-93d4-4556deb2118b_isrForCrosstalkSources_1125947_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:52 proc_lsst.multi:146 [INFO] found job 16896613 in provider checkpoint proc_lsst.multi INFO: found job 16896613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f24ff682f8e064ca64afbc9900393325 with result from task 3664 parsl.dataflow.dflow DEBUG: Task 3664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/f59a94ee-1e10-41eb-b082-57db1eed0fe6_isrForCrosstalkSources_1125918_48.stdout parsl.dataflow.dflow INFO: Standard error for task 3664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125918/f59a94ee-1e10-41eb-b082-57db1eed0fe6_isrForCrosstalkSources_1125918_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:53 proc_lsst.multi:146 [INFO] found job 16896614 in provider checkpoint proc_lsst.multi INFO: found job 16896614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:53 proc_lsst.multi:146 [INFO] found job 16896615 in provider checkpoint proc_lsst.multi INFO: found job 16896615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:53 proc_lsst.multi:146 [INFO] found job 16896616 in provider checkpoint proc_lsst.multi INFO: found job 16896616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:53 proc_lsst.multi:146 [INFO] found job 16896617 in provider checkpoint proc_lsst.multi INFO: found job 16896617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.dflow INFO: Task 3693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a7c86457df8289b5baf7880fb1efdde with result from task 3693 parsl.dataflow.dflow DEBUG: Task 3693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6d5cd192-18b4-4f32-b926-af889e776615_isrForCrosstalkSources_1125927_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125927/6d5cd192-18b4-4f32-b926-af889e776615_isrForCrosstalkSources_1125927_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:53 proc_lsst.multi:146 [INFO] found job 16896618 in provider checkpoint proc_lsst.multi INFO: found job 16896618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:54 proc_lsst.multi:146 [INFO] found job 16896619 in provider checkpoint proc_lsst.multi INFO: found job 16896619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:54 proc_lsst.multi:146 [INFO] found job 16896620 in provider checkpoint proc_lsst.multi INFO: found job 16896620 in provider checkpoint parsl.dataflow.dflow INFO: Task 3696 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.dataflow.memoization DEBUG: Storing app cache entry b2d12062b6ff56009437630a95356f5f with result from task 3696 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 3720 parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: Task 3696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/78953df3-3b44-4b01-a063-f4b96576c7a2_isrForCrosstalkSources_1125892_57.stdout parsl.dataflow.dflow INFO: Standard error for task 3696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/submit/DEEP/20220826/B1l/science#step0/20240307T150529Z/logs/isrForCrosstalkSources/1125892/78953df3-3b44-4b01-a063-f4b96576c7a2_isrForCrosstalkSources_1125892_57.stderr parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:54 proc_lsst.multi:146 [INFO] found job 16896621 in provider checkpoint proc_lsst.multi INFO: found job 16896621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:54 proc_lsst.multi:146 [INFO] found job 16896622 in provider checkpoint proc_lsst.multi INFO: found job 16896622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:54 proc_lsst.multi:146 [INFO] found job 16896623 in provider checkpoint proc_lsst.multi INFO: found job 16896623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:55 proc_lsst.multi:146 [INFO] found job 16896624 in provider checkpoint proc_lsst.multi INFO: found job 16896624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:55 proc_lsst.multi:146 [INFO] found job 16896625 in provider checkpoint proc_lsst.multi INFO: found job 16896625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:55 proc_lsst.multi:146 [INFO] found job 16896626 in provider checkpoint proc_lsst.multi INFO: found job 16896626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:55 proc_lsst.multi:146 [INFO] found job 16896627 in provider checkpoint proc_lsst.multi INFO: found job 16896627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:56 proc_lsst.multi:146 [INFO] found job 16896628 in provider checkpoint proc_lsst.multi INFO: found job 16896628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:56 proc_lsst.multi:146 [INFO] found job 16896629 in provider checkpoint proc_lsst.multi INFO: found job 16896629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:58 proc_lsst.multi:146 [INFO] found job 16896630 in provider checkpoint proc_lsst.multi INFO: found job 16896630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:58 proc_lsst.multi:146 [INFO] found job 16896631 in provider checkpoint proc_lsst.multi INFO: found job 16896631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896505 PD 16896504 PD 16896503 PD 16896502 PD 16896501 PD 16896500 PD 16896499 PD 16896498 PD 16896497 PD 16896496 PD 16896495 PD 16896494 PD 16896493 PD 16896492 PD 16896491 PD 16896490 PD 16896489 PD 16896488 PD 16896487 PD 16896486 PD 16896485 PD 16896484 PD 16896483 PD 16896482 PD 16896481 PD 16896480 PD 16896479 PD 16896478 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:58 proc_lsst.multi:146 [INFO] found job 16896632 in provider checkpoint proc_lsst.multi INFO: found job 16896632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:59 proc_lsst.multi:146 [INFO] found job 16896633 in provider checkpoint proc_lsst.multi INFO: found job 16896633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:59 proc_lsst.multi:146 [INFO] found job 16896634 in provider checkpoint proc_lsst.multi INFO: found job 16896634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:14:59 proc_lsst.multi:146 [INFO] found job 16896635 in provider checkpoint proc_lsst.multi INFO: found job 16896635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:00 proc_lsst.multi:146 [INFO] found job 16896636 in provider checkpoint proc_lsst.multi INFO: found job 16896636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:00 proc_lsst.multi:146 [INFO] found job 16896637 in provider checkpoint proc_lsst.multi INFO: found job 16896637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:00 proc_lsst.multi:146 [INFO] found job 16896638 in provider checkpoint proc_lsst.multi INFO: found job 16896638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:00 proc_lsst.multi:146 [INFO] found job 16896639 in provider checkpoint proc_lsst.multi INFO: found job 16896639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:00 proc_lsst.multi:146 [INFO] found job 16896640 in provider checkpoint proc_lsst.multi INFO: found job 16896640 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:01 proc_lsst.multi:146 [INFO] found job 16896641 in provider checkpoint proc_lsst.multi INFO: found job 16896641 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:01 proc_lsst.multi:146 [INFO] found job 16896642 in provider checkpoint proc_lsst.multi INFO: found job 16896642 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:01 proc_lsst.multi:146 [INFO] found job 16896643 in provider checkpoint proc_lsst.multi INFO: found job 16896643 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:01 proc_lsst.multi:146 [INFO] found job 16896644 in provider checkpoint proc_lsst.multi INFO: found job 16896644 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:01 proc_lsst.multi:146 [INFO] found job 16896645 in provider checkpoint proc_lsst.multi INFO: found job 16896645 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:02 proc_lsst.multi:146 [INFO] found job 16896646 in provider checkpoint proc_lsst.multi INFO: found job 16896646 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:02 proc_lsst.multi:146 [INFO] found job 16896647 in provider checkpoint proc_lsst.multi INFO: found job 16896647 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:02 proc_lsst.multi:146 [INFO] found job 16896648 in provider checkpoint proc_lsst.multi INFO: found job 16896648 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:02 proc_lsst.multi:146 [INFO] found job 16896649 in provider checkpoint proc_lsst.multi INFO: found job 16896649 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:03 proc_lsst.multi:146 [INFO] found job 16896650 in provider checkpoint proc_lsst.multi INFO: found job 16896650 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:03 proc_lsst.multi:146 [INFO] found job 16896651 in provider checkpoint proc_lsst.multi INFO: found job 16896651 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:03 proc_lsst.multi:146 [INFO] found job 16896652 in provider checkpoint proc_lsst.multi INFO: found job 16896652 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:03 proc_lsst.multi:146 [INFO] found job 16896653 in provider checkpoint proc_lsst.multi INFO: found job 16896653 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:03 proc_lsst.multi:146 [INFO] found job 16896654 in provider checkpoint proc_lsst.multi INFO: found job 16896654 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:04 proc_lsst.multi:146 [INFO] found job 16896655 in provider checkpoint proc_lsst.multi INFO: found job 16896655 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:04 proc_lsst.multi:146 [INFO] found job 16896656 in provider checkpoint proc_lsst.multi INFO: found job 16896656 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:04 proc_lsst.multi:146 [INFO] found job 16896657 in provider checkpoint proc_lsst.multi INFO: found job 16896657 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:04 proc_lsst.multi:146 [INFO] found job 16896658 in provider checkpoint proc_lsst.multi INFO: found job 16896658 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:04 proc_lsst.multi:146 [INFO] found job 16896659 in provider checkpoint proc_lsst.multi INFO: found job 16896659 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:05 proc_lsst.multi:146 [INFO] found job 16896660 in provider checkpoint proc_lsst.multi INFO: found job 16896660 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:05 proc_lsst.multi:146 [INFO] found job 16896661 in provider checkpoint proc_lsst.multi INFO: found job 16896661 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:05 proc_lsst.multi:146 [INFO] found job 16896662 in provider checkpoint proc_lsst.multi INFO: found job 16896662 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:05 proc_lsst.multi:146 [INFO] found job 16896663 in provider checkpoint proc_lsst.multi INFO: found job 16896663 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:05 proc_lsst.multi:146 [INFO] found job 16896664 in provider checkpoint proc_lsst.multi INFO: found job 16896664 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:06 proc_lsst.multi:146 [INFO] found job 16896665 in provider checkpoint proc_lsst.multi INFO: found job 16896665 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:06 proc_lsst.multi:146 [INFO] found job 16896666 in provider checkpoint proc_lsst.multi INFO: found job 16896666 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:06 proc_lsst.multi:146 [INFO] found job 16896667 in provider checkpoint proc_lsst.multi INFO: found job 16896667 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:06 proc_lsst.multi:146 [INFO] found job 16896668 in provider checkpoint proc_lsst.multi INFO: found job 16896668 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:07 proc_lsst.multi:146 [INFO] found job 16896669 in provider checkpoint proc_lsst.multi INFO: found job 16896669 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:07 proc_lsst.multi:146 [INFO] found job 16896670 in provider checkpoint proc_lsst.multi INFO: found job 16896670 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:07 proc_lsst.multi:146 [INFO] found job 16896671 in provider checkpoint proc_lsst.multi INFO: found job 16896671 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:07 proc_lsst.multi:146 [INFO] found job 16896672 in provider checkpoint proc_lsst.multi INFO: found job 16896672 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:07 proc_lsst.multi:146 [INFO] found job 16896673 in provider checkpoint proc_lsst.multi INFO: found job 16896673 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:08 proc_lsst.multi:146 [INFO] found job 16896674 in provider checkpoint proc_lsst.multi INFO: found job 16896674 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:08 proc_lsst.multi:146 [INFO] found job 16896675 in provider checkpoint proc_lsst.multi INFO: found job 16896675 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:08 proc_lsst.multi:146 [INFO] found job 16896676 in provider checkpoint proc_lsst.multi INFO: found job 16896676 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:08 proc_lsst.multi:146 [INFO] found job 16896678 in provider checkpoint proc_lsst.multi INFO: found job 16896678 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:08 proc_lsst.multi:146 [INFO] found job 16896679 in provider checkpoint proc_lsst.multi INFO: found job 16896679 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:09 proc_lsst.multi:146 [INFO] found job 16896680 in provider checkpoint proc_lsst.multi INFO: found job 16896680 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:09 proc_lsst.multi:146 [INFO] found job 16896681 in provider checkpoint proc_lsst.multi INFO: found job 16896681 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:09 proc_lsst.multi:146 [INFO] found job 16896682 in provider checkpoint proc_lsst.multi INFO: found job 16896682 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:09 proc_lsst.multi:146 [INFO] found job 16896683 in provider checkpoint proc_lsst.multi INFO: found job 16896683 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:10 proc_lsst.multi:146 [INFO] found job 16896684 in provider checkpoint proc_lsst.multi INFO: found job 16896684 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:10 proc_lsst.multi:146 [INFO] found job 16896685 in provider checkpoint proc_lsst.multi INFO: found job 16896685 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:10 proc_lsst.multi:146 [INFO] found job 16896686 in provider checkpoint proc_lsst.multi INFO: found job 16896686 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:10 proc_lsst.multi:146 [INFO] found job 16896687 in provider checkpoint proc_lsst.multi INFO: found job 16896687 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:10 proc_lsst.multi:146 [INFO] found job 16896688 in provider checkpoint proc_lsst.multi INFO: found job 16896688 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:11 proc_lsst.multi:146 [INFO] found job 16896689 in provider checkpoint proc_lsst.multi INFO: found job 16896689 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:11 proc_lsst.multi:146 [INFO] found job 16896690 in provider checkpoint proc_lsst.multi INFO: found job 16896690 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:11 proc_lsst.multi:146 [INFO] found job 16896691 in provider checkpoint proc_lsst.multi INFO: found job 16896691 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:11 proc_lsst.multi:146 [INFO] found job 16896692 in provider checkpoint proc_lsst.multi INFO: found job 16896692 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:11 proc_lsst.multi:146 [INFO] found job 16896693 in provider checkpoint proc_lsst.multi INFO: found job 16896693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:12 proc_lsst.multi:146 [INFO] found job 16896694 in provider checkpoint proc_lsst.multi INFO: found job 16896694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:12 proc_lsst.multi:146 [INFO] found job 16896695 in provider checkpoint proc_lsst.multi INFO: found job 16896695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:12 proc_lsst.multi:146 [INFO] found job 16896696 in provider checkpoint proc_lsst.multi INFO: found job 16896696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:12 proc_lsst.multi:146 [INFO] found job 16896697 in provider checkpoint proc_lsst.multi INFO: found job 16896697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:13 proc_lsst.multi:146 [INFO] found job 16896698 in provider checkpoint proc_lsst.multi INFO: found job 16896698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:13 proc_lsst.multi:146 [INFO] found job 16896699 in provider checkpoint proc_lsst.multi INFO: found job 16896699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:13 proc_lsst.multi:146 [INFO] found job 16896700 in provider checkpoint proc_lsst.multi INFO: found job 16896700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:13 proc_lsst.multi:146 [INFO] found job 16896701 in provider checkpoint proc_lsst.multi INFO: found job 16896701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:13 proc_lsst.multi:146 [INFO] found job 16896702 in provider checkpoint proc_lsst.multi INFO: found job 16896702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:14 proc_lsst.multi:146 [INFO] found job 16896703 in provider checkpoint proc_lsst.multi INFO: found job 16896703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:14 proc_lsst.multi:146 [INFO] found job 16896704 in provider checkpoint proc_lsst.multi INFO: found job 16896704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:14 proc_lsst.multi:146 [INFO] found job 16896705 in provider checkpoint proc_lsst.multi INFO: found job 16896705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:14 proc_lsst.multi:146 [INFO] found job 16896706 in provider checkpoint proc_lsst.multi INFO: found job 16896706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:14 proc_lsst.multi:146 [INFO] found job 16896707 in provider checkpoint proc_lsst.multi INFO: found job 16896707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:15 proc_lsst.multi:146 [INFO] found job 16896708 in provider checkpoint proc_lsst.multi INFO: found job 16896708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:15 proc_lsst.multi:146 [INFO] found job 16896709 in provider checkpoint proc_lsst.multi INFO: found job 16896709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:15 proc_lsst.multi:146 [INFO] found job 16896710 in provider checkpoint proc_lsst.multi INFO: found job 16896710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:15 proc_lsst.multi:146 [INFO] found job 16896711 in provider checkpoint proc_lsst.multi INFO: found job 16896711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:16 proc_lsst.multi:146 [INFO] found job 16896712 in provider checkpoint proc_lsst.multi INFO: found job 16896712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:16 proc_lsst.multi:146 [INFO] found job 16896713 in provider checkpoint proc_lsst.multi INFO: found job 16896713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING 2024-03-07 15:15:16 proc_lsst.multi:146 [INFO] found job 16896714 in provider checkpoint proc_lsst.multi INFO: found job 16896714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16896445,16896446,16896447,16896448,16896449,16896450,16896451,16896452,16896453,16896454,16896455,16896456,16896457,16896458,16896459,16896460,16896461,16896462,16896463,16896464,16896466,16896467,16896468,16896469,16896470,16896471,16896472,16896473,16896474,16896475,16896476,16896477,16896478,16896479,16896480,16896481,16896482,16896483,16896484,16896485,16896486,16896487,16896488,16896489,16896490,16896491,16896492,16896493,16896494,16896495,16896496,16896497,16896498,16896499,16896500,16896501,16896502,16896503,16896504,16896505,16896506,16896507,16896508,16896509,16896510,16896511,16896519,16896520,16896521,16896522,16896523,16896524,16896525,16896526,16896527,16896528,16896529,16896530,16896531,16896533,16896534,16896535,16896536,16896537,16896538,16896539,16896540,16896541,16896542,16896543,16896544,16896545,16896546,16896547,16896548,16896549,16896550,16896551,16896552,16896553,16896554,16896555,16896556,16896559,16896561,16896562,16896563,16896564,16896565,16896566,16896568,16896569,16896570,16896571,16896572,16896573,16896574,16896575,16896576,16896577,16896578,16896579,16896580,16896581,16896582,16896583,16896584,16896585,16896586,16896587,16896588,16896589,16896590,16896591,16896592,16896593,16896594,16896595,16896596,16896597,16896598,16896599,16896600,16896601,16896602,16896603,16896604,16896605,16896606,16896607,16896608,16896609,16896610,16896611,16896612,16896613,16896614,16896615,16896616,16896617,16896618,16896619,16896620,16896621,16896622,16896623,16896624,16896625,16896626,16896627,16896628,16896629,16896630,16896631,16896632,16896633,16896634,16896635,16896636,16896637,16896638,16896639,16896640,16896641,16896642,16896643,16896644,16896645,16896646,16896647,16896648,16896649,16896650,16896651,16896652,16896653,16896654,16896655,16896656,16896657,16896658,16896659,16896660,16896661,16896662,16896663,16896664,16896665,16896666,16896667,16896668,16896669,16896670,16896671,16896672,16896673,16896674,16896675,16896676,16896678,16896679,16896680,16896681,16896682,16896683,16896684,16896685,16896686,16896687,16896688,16896689,16896690,16896691,16896692,16896693,16896694,16896695,16896696,16896697,16896698,16896699,16896700,16896701,16896702,16896703,16896704,16896705,16896706,16896707,16896708,16896709,16896710,16896711,16896712,16896713,16896714' parsl.providers.slurm.slurm DEBUG: squeue returned 16896478 PD 16896479 PD 16896480 PD 16896481 PD 16896482 PD 16896483 PD 16896484 PD 16896485 PD 16896486 PD 16896487 PD 16896488 PD 16896489 PD 16896490 PD 16896491 PD 16896492 PD 16896493 PD 16896494 PD 16896495 PD 16896496 PD 16896497 PD 16896498 PD 16896499 PD 16896500 PD 16896501 PD 16896502 PD 16896503 PD 16896504 PD 16896505 PD 16896714 PD 16896713 PD 16896712 PD 16896711 PD 16896710 PD 16896709 PD 16896708 PD 16896707 PD 16896706 PD 16896705 PD 16896704 PD 16896703 PD 16896702 PD 16896701 PD 16896700 PD 16896699 PD 16896698 PD 16896697 PD 16896696 PD 16896695 PD 16896694 PD 16896693 PD 16896692 PD 16896691 PD 16896690 PD 16896689 PD 16896688 PD 16896687 PD 16896686 PD 16896685 PD 16896684 PD 16896683 PD 16896682 PD 16896681 PD 16896680 PD 16896679 PD 16896678 PD 16896676 PD 16896675 PD 16896674 PD 16896673 PD 16896672 PD 16896671 PD 16896670 PD 16896669 PD 16896668 PD 16896667 PD 16896666 PD 16896665 PD 16896664 PD 16896663 PD 16896662 PD 16896661 PD 16896660 PD 16896659 PD 16896658 PD 16896657 PD 16896656 PD 16896655 PD 16896654 PD 16896653 PD 16896652 PD 16896651 PD 16896650 PD 16896649 PD 16896648 PD 16896647 PD 16896646 PD 16896645 PD 16896644 PD 16896643 PD 16896642 PD 16896641 PD 16896640 PD 16896639 PD 16896638 PD 16896637 PD 16896636 PD 16896635 PD 16896634 PD 16896633 PD 16896632 PD 16896631 PD 16896630 PD 16896629 PD 16896628 PD 16896627 PD 16896626 PD 16896625 PD 16896624 PD 16896623 PD 16896622 PD 16896621 PD 16896620 PD 16896619 PD 16896618 PD 16896617 PD 16896616 PD 16896615 PD 16896614 PD 16896613 PD 16896612 PD 16896611 PD 16896610 PD 16896609 PD 16896608 PD 16896607 PD 16896606 PD 16896605 PD 16896604 PD 16896603 PD 16896602 PD 16896601 PD 16896600 PD 16896599 PD 16896598 PD 16896597 PD 16896596 PD 16896595 PD 16896594 PD 16896593 PD 16896592 PD 16896591 PD 16896590 PD 16896589 PD 16896588 PD 16896587 PD 16896586 PD 16896585 PD 16896584 PD 16896583 PD 16896582 PD 16896581 PD 16896580 PD 16896579 PD 16896578 PD 16896577 PD 16896576 PD 16896575 PD 16896574 PD 16896573 PD 16896572 PD 16896571 PD 16896570 PD 16896569 PD 16896568 PD 16896566 PD 16896565 PD 16896564 PD 16896563 PD 16896562 PD 16896561 PD 16896559 PD 16896556 PD 16896555 PD 16896554 PD 16896553 PD 16896552 PD 16896551 PD 16896550 PD 16896549 PD 16896548 PD 16896547 PD 16896546 PD 16896545 PD 16896544 PD 16896543 PD 16896542 PD 16896541 PD 16896540 PD 16896539 PD 16896538 PD 16896537 PD 16896536 PD 16896535 PD 16896534 PD 16896533 PD 16896531 PD 16896530 PD 16896529 PD 16896528 PD 16896527 PD 16896526 PD 16896525 PD 16896524 PD 16896523 PD 16896522 PD 16896521 PD 16896520 PD 16896519 PD 16896511 PD 16896510 PD 16896509 PD 16896508 PD 16896507 PD 16896506 PD 16896449 R 16896450 R 16896451 R 16896452 R 16896453 R 16896454 R 16896455 R 16896456 R 16896457 R 16896458 R 16896459 R 16896460 R 16896461 R 16896462 R 16896463 R 16896464 R 16896466 R 16896467 R 16896468 R 16896469 R 16896470 R 16896471 R 16896472 R 16896473 R 16896474 R 16896475 R 16896476 R 16896477 R 16896447 R 16896448 R 16896446 R 16896445 R parsl.providers.slurm.slurm DEBUG: Updating job 16896478 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896479 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896480 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896481 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896482 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896483 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896484 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896485 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896486 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896487 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896488 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896489 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896490 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896491 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896492 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896493 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896494 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896495 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896496 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896497 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896498 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896499 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896500 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896501 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896502 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896503 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896504 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896505 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896714 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896713 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896712 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896711 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896710 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896709 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896708 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896707 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896706 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896705 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896704 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896703 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896702 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896701 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896700 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896699 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896698 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896697 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896696 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896695 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896694 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896693 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896692 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896691 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896689 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896688 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896687 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896686 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896685 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896684 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896683 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896682 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896681 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896680 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896679 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896678 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896676 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896675 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896674 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896673 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896672 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896671 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896670 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896669 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896668 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896667 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896666 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896665 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896664 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896663 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896662 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896661 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896660 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896659 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896658 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896657 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896656 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896655 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896654 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896653 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896652 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896651 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896650 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896649 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896648 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896647 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896646 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896645 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896644 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896643 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896642 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896641 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896640 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896639 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896638 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896637 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896636 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896635 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896634 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896633 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896632 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896631 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896630 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896629 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896628 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896627 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896626 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896625 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896624 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896623 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896622 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896621 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896620 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896619 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896618 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896617 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896616 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896615 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896614 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896613 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896612 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896611 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896610 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896609 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896608 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896607 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896606 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896603 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896602 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896601 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896600 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896599 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896598 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896597 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896596 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896595 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896594 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896593 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896592 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896591 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896590 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896589 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896588 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896587 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896586 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896585 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896584 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896583 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896582 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896581 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896580 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896579 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896578 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896577 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896576 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896575 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896574 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896573 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896572 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896571 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896570 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896569 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896568 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896566 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896565 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896564 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896563 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896562 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896561 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896559 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896556 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896555 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896554 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896553 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896552 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896551 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896550 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896549 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896548 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896547 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896546 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896545 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896544 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896543 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896542 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896541 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896540 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896539 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896538 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896537 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896536 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896535 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896534 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896533 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896531 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896530 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896529 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896528 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896527 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896526 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896525 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896524 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896523 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896522 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896521 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896520 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896519 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896511 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896510 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896509 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896508 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896507 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896506 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16896449 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896450 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896451 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896452 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896453 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896454 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896455 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896456 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896457 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896458 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896459 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896460 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896461 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896462 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896463 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896464 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896466 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896467 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896468 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896469 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896470 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896471 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896472 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896473 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896474 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896475 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896476 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896477 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896447 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896448 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896446 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16896445 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 312, active_tasks = 0 parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 66/246 running/pending blocks, and 291 connected workers parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (312) than minimum blocks (0) parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in parsl.dataflow.strategy DEBUG: Idle time 7.152557373046875e-07s is less than max_idletime 120.0s for executor multi; not scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22377792571472 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=312, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 66 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 78ce42f63c09 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 78ce42f63c09 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 24a6abe3cc7c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 24a6abe3cc7c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4e9b790a00e3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4e9b790a00e3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4df237094912 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4df237094912 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0b06e350d6e2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0b06e350d6e2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fe0b85f90633 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fe0b85f90633 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aff2eb4cc254 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aff2eb4cc254 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f0186f526273 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f0186f526273 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e2e1e0f25d2c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e2e1e0f25d2c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 75fce2b11d29 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 75fce2b11d29 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d2b8844c718f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d2b8844c718f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 02a7ce557f8b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 02a7ce557f8b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1f0a922f4967 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1f0a922f4967 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 643975e78b5b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 643975e78b5b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 458d62de9c37 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 458d62de9c37 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a54944f53aef parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a54944f53aef parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2f43327bb796 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2f43327bb796 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b94f9ffb4bab parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b94f9ffb4bab parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e386e1d3cc08 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e386e1d3cc08 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 061bcd3eaa19 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 061bcd3eaa19 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f7d7e34407d8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f7d7e34407d8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c07a073cb163 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c07a073cb163 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 453c548a6f32 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 453c548a6f32 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 926f547d84ec parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 926f547d84ec parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e0cc2cfad43d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e0cc2cfad43d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6988deb4a817 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6988deb4a817 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f7c59eea9094 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f7c59eea9094 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5fb1c3d4a77c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5fb1c3d4a77c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 75659ccb1ac6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 75659ccb1ac6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8bd1f1595584 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8bd1f1595584 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6c33acdc5b42 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6c33acdc5b42 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6df869a8cbab parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6df869a8cbab parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 18a263126c4b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 18a263126c4b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 47f4a51defee parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 47f4a51defee parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aa8608aa1d9c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aa8608aa1d9c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4795feeb2531 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4795feeb2531 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b80babc89175 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b80babc89175 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 00e592cd0f73 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 00e592cd0f73 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d0a3313e9a4f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d0a3313e9a4f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8fb8add9aa46 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8fb8add9aa46 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cb5027e556c3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cb5027e556c3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e270c32114b4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e270c32114b4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4f23ca3b6c17 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4f23ca3b6c17 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 62195c7ae924 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 62195c7ae924 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 62816070060c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 62816070060c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fafa1b124d8f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fafa1b124d8f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9d541ee63174 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9d541ee63174 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3fdc4fdb9827 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3fdc4fdb9827 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 104e9130ee64 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 104e9130ee64 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ef0e3d1328cd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ef0e3d1328cd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 91f48a1e56bd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 91f48a1e56bd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 18da92042bd3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 18da92042bd3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 880845f8a657 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 880845f8a657 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 67201e1b8df6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 67201e1b8df6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5d708c8e0e4b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5d708c8e0e4b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2bfbabfda08a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2bfbabfda08a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 320290c74b77 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 320290c74b77 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cc327e59efd4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cc327e59efd4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3722b7f2d638 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3722b7f2d638 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 377d4538cfa7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 377d4538cfa7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 621c7ba27905 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 621c7ba27905 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8608917104ee parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8608917104ee parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d640c105b8d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d640c105b8d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f9872edd60a5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f9872edd60a5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c79f49e11225 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c79f49e11225 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 920b28a6c43b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 920b28a6c43b 2024-03-07 15:15:16 proc_lsst.multi:146 [INFO] found job 16896469 in provider checkpoint proc_lsst.multi INFO: found job 16896469 in provider checkpoint 2024-03-07 15:15:16 proc_lsst.multi:201 [INFO] cancelling 16896469 on provider checkpoint proc_lsst.multi INFO: cancelling 16896469 on provider checkpoint 2024-03-07 15:15:16 proc_lsst.multi:146 [INFO] found job 16896435 in provider escience_a40 proc_lsst.multi INFO: found job 16896435 in provider escience_a40 2024-03-07 15:15:16 proc_lsst.multi:201 [INFO] cancelling 16896435 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896435 on provider escience_a40 2024-03-07 15:15:17 proc_lsst.multi:146 [INFO] found job 16896434 in provider escience_a40 proc_lsst.multi INFO: found job 16896434 in provider escience_a40 2024-03-07 15:15:17 proc_lsst.multi:201 [INFO] cancelling 16896434 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896434 on provider escience_a40 2024-03-07 15:15:17 proc_lsst.multi:146 [INFO] found job 16896433 in provider escience_a40 proc_lsst.multi INFO: found job 16896433 in provider escience_a40 2024-03-07 15:15:17 proc_lsst.multi:201 [INFO] cancelling 16896433 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896433 on provider escience_a40 2024-03-07 15:15:17 proc_lsst.multi:146 [INFO] found job 16896447 in provider checkpoint proc_lsst.multi INFO: found job 16896447 in provider checkpoint 2024-03-07 15:15:17 proc_lsst.multi:201 [INFO] cancelling 16896447 on provider checkpoint proc_lsst.multi INFO: cancelling 16896447 on provider checkpoint 2024-03-07 15:15:17 proc_lsst.multi:146 [INFO] found job 16896466 in provider checkpoint proc_lsst.multi INFO: found job 16896466 in provider checkpoint 2024-03-07 15:15:17 proc_lsst.multi:201 [INFO] cancelling 16896466 on provider checkpoint proc_lsst.multi INFO: cancelling 16896466 on provider checkpoint 2024-03-07 15:15:17 proc_lsst.multi:146 [INFO] found job 16896460 in provider checkpoint proc_lsst.multi INFO: found job 16896460 in provider checkpoint 2024-03-07 15:15:17 proc_lsst.multi:201 [INFO] cancelling 16896460 on provider checkpoint proc_lsst.multi INFO: cancelling 16896460 on provider checkpoint 2024-03-07 15:15:17 proc_lsst.multi:146 [INFO] found job 16896432 in provider escience_a40 proc_lsst.multi INFO: found job 16896432 in provider escience_a40 2024-03-07 15:15:17 proc_lsst.multi:201 [INFO] cancelling 16896432 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896432 on provider escience_a40 2024-03-07 15:15:17 proc_lsst.multi:146 [INFO] found job 16896461 in provider checkpoint proc_lsst.multi INFO: found job 16896461 in provider checkpoint 2024-03-07 15:15:17 proc_lsst.multi:201 [INFO] cancelling 16896461 on provider checkpoint proc_lsst.multi INFO: cancelling 16896461 on provider checkpoint 2024-03-07 15:15:17 proc_lsst.multi:146 [INFO] found job 16896474 in provider checkpoint proc_lsst.multi INFO: found job 16896474 in provider checkpoint 2024-03-07 15:15:17 proc_lsst.multi:201 [INFO] cancelling 16896474 on provider checkpoint proc_lsst.multi INFO: cancelling 16896474 on provider checkpoint 2024-03-07 15:15:17 proc_lsst.multi:146 [INFO] found job 16896457 in provider checkpoint proc_lsst.multi INFO: found job 16896457 in provider checkpoint 2024-03-07 15:15:17 proc_lsst.multi:201 [INFO] cancelling 16896457 on provider checkpoint proc_lsst.multi INFO: cancelling 16896457 on provider checkpoint 2024-03-07 15:15:17 proc_lsst.multi:146 [INFO] found job 16896475 in provider checkpoint proc_lsst.multi INFO: found job 16896475 in provider checkpoint 2024-03-07 15:15:17 proc_lsst.multi:201 [INFO] cancelling 16896475 on provider checkpoint proc_lsst.multi INFO: cancelling 16896475 on provider checkpoint 2024-03-07 15:15:18 proc_lsst.multi:146 [INFO] found job 16896464 in provider checkpoint proc_lsst.multi INFO: found job 16896464 in provider checkpoint 2024-03-07 15:15:18 proc_lsst.multi:201 [INFO] cancelling 16896464 on provider checkpoint proc_lsst.multi INFO: cancelling 16896464 on provider checkpoint 2024-03-07 15:15:18 proc_lsst.multi:146 [INFO] found job 16896471 in provider checkpoint proc_lsst.multi INFO: found job 16896471 in provider checkpoint 2024-03-07 15:15:18 proc_lsst.multi:201 [INFO] cancelling 16896471 on provider checkpoint proc_lsst.multi INFO: cancelling 16896471 on provider checkpoint 2024-03-07 15:15:18 proc_lsst.multi:146 [INFO] found job 16896390 in provider astro proc_lsst.multi INFO: found job 16896390 in provider astro 2024-03-07 15:15:18 proc_lsst.multi:201 [INFO] cancelling 16896390 on provider astro proc_lsst.multi INFO: cancelling 16896390 on provider astro 2024-03-07 15:15:18 proc_lsst.multi:146 [INFO] found job 16896385 in provider astro proc_lsst.multi INFO: found job 16896385 in provider astro 2024-03-07 15:15:18 proc_lsst.multi:201 [INFO] cancelling 16896385 on provider astro proc_lsst.multi INFO: cancelling 16896385 on provider astro 2024-03-07 15:15:18 proc_lsst.multi:146 [INFO] found job 16896472 in provider checkpoint proc_lsst.multi INFO: found job 16896472 in provider checkpoint 2024-03-07 15:15:18 proc_lsst.multi:201 [INFO] cancelling 16896472 on provider checkpoint proc_lsst.multi INFO: cancelling 16896472 on provider checkpoint 2024-03-07 15:15:18 proc_lsst.multi:146 [INFO] found job 16896462 in provider checkpoint proc_lsst.multi INFO: found job 16896462 in provider checkpoint 2024-03-07 15:15:18 proc_lsst.multi:201 [INFO] cancelling 16896462 on provider checkpoint proc_lsst.multi INFO: cancelling 16896462 on provider checkpoint 2024-03-07 15:15:18 proc_lsst.multi:146 [INFO] found job 16896470 in provider checkpoint proc_lsst.multi INFO: found job 16896470 in provider checkpoint 2024-03-07 15:15:18 proc_lsst.multi:201 [INFO] cancelling 16896470 on provider checkpoint proc_lsst.multi INFO: cancelling 16896470 on provider checkpoint 2024-03-07 15:15:18 proc_lsst.multi:146 [INFO] found job 16896392 in provider astro proc_lsst.multi INFO: found job 16896392 in provider astro 2024-03-07 15:15:18 proc_lsst.multi:201 [INFO] cancelling 16896392 on provider astro proc_lsst.multi INFO: cancelling 16896392 on provider astro 2024-03-07 15:15:18 proc_lsst.multi:146 [INFO] found job 16896459 in provider checkpoint proc_lsst.multi INFO: found job 16896459 in provider checkpoint 2024-03-07 15:15:18 proc_lsst.multi:201 [INFO] cancelling 16896459 on provider checkpoint proc_lsst.multi INFO: cancelling 16896459 on provider checkpoint 2024-03-07 15:15:19 proc_lsst.multi:146 [INFO] found job 16896446 in provider checkpoint proc_lsst.multi INFO: found job 16896446 in provider checkpoint 2024-03-07 15:15:19 proc_lsst.multi:201 [INFO] cancelling 16896446 on provider checkpoint proc_lsst.multi INFO: cancelling 16896446 on provider checkpoint 2024-03-07 15:15:19 proc_lsst.multi:146 [INFO] found job 16896455 in provider checkpoint proc_lsst.multi INFO: found job 16896455 in provider checkpoint 2024-03-07 15:15:19 proc_lsst.multi:201 [INFO] cancelling 16896455 on provider checkpoint proc_lsst.multi INFO: cancelling 16896455 on provider checkpoint 2024-03-07 15:15:19 proc_lsst.multi:146 [INFO] found job 16896388 in provider astro proc_lsst.multi INFO: found job 16896388 in provider astro 2024-03-07 15:15:19 proc_lsst.multi:201 [INFO] cancelling 16896388 on provider astro proc_lsst.multi INFO: cancelling 16896388 on provider astro 2024-03-07 15:15:19 proc_lsst.multi:146 [INFO] found job 16896473 in provider checkpoint proc_lsst.multi INFO: found job 16896473 in provider checkpoint 2024-03-07 15:15:19 proc_lsst.multi:201 [INFO] cancelling 16896473 on provider checkpoint proc_lsst.multi INFO: cancelling 16896473 on provider checkpoint 2024-03-07 15:15:19 proc_lsst.multi:146 [INFO] found job 16896456 in provider checkpoint proc_lsst.multi INFO: found job 16896456 in provider checkpoint 2024-03-07 15:15:19 proc_lsst.multi:201 [INFO] cancelling 16896456 on provider checkpoint proc_lsst.multi INFO: cancelling 16896456 on provider checkpoint 2024-03-07 15:15:19 proc_lsst.multi:146 [INFO] found job 16896458 in provider checkpoint proc_lsst.multi INFO: found job 16896458 in provider checkpoint 2024-03-07 15:15:19 proc_lsst.multi:201 [INFO] cancelling 16896458 on provider checkpoint proc_lsst.multi INFO: cancelling 16896458 on provider checkpoint 2024-03-07 15:15:19 proc_lsst.multi:146 [INFO] found job 16896449 in provider checkpoint proc_lsst.multi INFO: found job 16896449 in provider checkpoint 2024-03-07 15:15:19 proc_lsst.multi:201 [INFO] cancelling 16896449 on provider checkpoint proc_lsst.multi INFO: cancelling 16896449 on provider checkpoint 2024-03-07 15:15:19 proc_lsst.multi:146 [INFO] found job 16896454 in provider checkpoint proc_lsst.multi INFO: found job 16896454 in provider checkpoint 2024-03-07 15:15:19 proc_lsst.multi:201 [INFO] cancelling 16896454 on provider checkpoint proc_lsst.multi INFO: cancelling 16896454 on provider checkpoint 2024-03-07 15:15:19 proc_lsst.multi:146 [INFO] found job 16896445 in provider checkpoint proc_lsst.multi INFO: found job 16896445 in provider checkpoint 2024-03-07 15:15:19 proc_lsst.multi:201 [INFO] cancelling 16896445 on provider checkpoint proc_lsst.multi INFO: cancelling 16896445 on provider checkpoint 2024-03-07 15:15:20 proc_lsst.multi:146 [INFO] found job 16896437 in provider escience_a40 proc_lsst.multi INFO: found job 16896437 in provider escience_a40 2024-03-07 15:15:20 proc_lsst.multi:201 [INFO] cancelling 16896437 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896437 on provider escience_a40 2024-03-07 15:15:20 proc_lsst.multi:146 [INFO] found job 16896441 in provider escience_a40 proc_lsst.multi INFO: found job 16896441 in provider escience_a40 2024-03-07 15:15:20 proc_lsst.multi:201 [INFO] cancelling 16896441 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896441 on provider escience_a40 2024-03-07 15:15:20 proc_lsst.multi:146 [INFO] found job 16896419 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896419 in provider escience_rtx6k 2024-03-07 15:15:20 proc_lsst.multi:201 [INFO] cancelling 16896419 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16896419 on provider escience_rtx6k 2024-03-07 15:15:20 proc_lsst.multi:146 [INFO] found job 16896443 in provider escience_a40 proc_lsst.multi INFO: found job 16896443 in provider escience_a40 2024-03-07 15:15:20 proc_lsst.multi:201 [INFO] cancelling 16896443 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896443 on provider escience_a40 2024-03-07 15:15:20 proc_lsst.multi:146 [INFO] found job 16896439 in provider escience_a40 proc_lsst.multi INFO: found job 16896439 in provider escience_a40 2024-03-07 15:15:20 proc_lsst.multi:201 [INFO] cancelling 16896439 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896439 on provider escience_a40 2024-03-07 15:15:20 proc_lsst.multi:146 [INFO] found job 16896444 in provider escience_a40 proc_lsst.multi INFO: found job 16896444 in provider escience_a40 2024-03-07 15:15:20 proc_lsst.multi:201 [INFO] cancelling 16896444 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896444 on provider escience_a40 2024-03-07 15:15:20 proc_lsst.multi:146 [INFO] found job 16896442 in provider escience_a40 proc_lsst.multi INFO: found job 16896442 in provider escience_a40 2024-03-07 15:15:20 proc_lsst.multi:201 [INFO] cancelling 16896442 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896442 on provider escience_a40 2024-03-07 15:15:20 proc_lsst.multi:146 [INFO] found job 16896467 in provider checkpoint proc_lsst.multi INFO: found job 16896467 in provider checkpoint 2024-03-07 15:15:20 proc_lsst.multi:201 [INFO] cancelling 16896467 on provider checkpoint proc_lsst.multi INFO: cancelling 16896467 on provider checkpoint 2024-03-07 15:15:20 proc_lsst.multi:146 [INFO] found job 16896438 in provider escience_a40 proc_lsst.multi INFO: found job 16896438 in provider escience_a40 2024-03-07 15:15:20 proc_lsst.multi:201 [INFO] cancelling 16896438 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896438 on provider escience_a40 2024-03-07 15:15:21 proc_lsst.multi:146 [INFO] found job 16896463 in provider checkpoint proc_lsst.multi INFO: found job 16896463 in provider checkpoint 2024-03-07 15:15:21 proc_lsst.multi:201 [INFO] cancelling 16896463 on provider checkpoint proc_lsst.multi INFO: cancelling 16896463 on provider checkpoint 2024-03-07 15:15:21 proc_lsst.multi:146 [INFO] found job 16896440 in provider escience_a40 proc_lsst.multi INFO: found job 16896440 in provider escience_a40 2024-03-07 15:15:21 proc_lsst.multi:201 [INFO] cancelling 16896440 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896440 on provider escience_a40 2024-03-07 15:15:21 proc_lsst.multi:146 [INFO] found job 16896468 in provider checkpoint proc_lsst.multi INFO: found job 16896468 in provider checkpoint 2024-03-07 15:15:21 proc_lsst.multi:201 [INFO] cancelling 16896468 on provider checkpoint proc_lsst.multi INFO: cancelling 16896468 on provider checkpoint 2024-03-07 15:15:21 proc_lsst.multi:146 [INFO] found job 16896476 in provider checkpoint proc_lsst.multi INFO: found job 16896476 in provider checkpoint 2024-03-07 15:15:21 proc_lsst.multi:201 [INFO] cancelling 16896476 on provider checkpoint proc_lsst.multi INFO: cancelling 16896476 on provider checkpoint 2024-03-07 15:15:21 proc_lsst.multi:146 [INFO] found job 53087 in provider local proc_lsst.multi INFO: found job 53087 in provider local 2024-03-07 15:15:21 proc_lsst.multi:201 [INFO] cancelling 53087 on provider local proc_lsst.multi INFO: cancelling 53087 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 53087 2024-03-07 15:15:21 proc_lsst.multi:146 [INFO] found job 16896477 in provider checkpoint proc_lsst.multi INFO: found job 16896477 in provider checkpoint 2024-03-07 15:15:21 proc_lsst.multi:201 [INFO] cancelling 16896477 on provider checkpoint proc_lsst.multi INFO: cancelling 16896477 on provider checkpoint 2024-03-07 15:15:21 proc_lsst.multi:146 [INFO] found job 16896448 in provider checkpoint proc_lsst.multi INFO: found job 16896448 in provider checkpoint 2024-03-07 15:15:21 proc_lsst.multi:201 [INFO] cancelling 16896448 on provider checkpoint proc_lsst.multi INFO: cancelling 16896448 on provider checkpoint 2024-03-07 15:15:21 proc_lsst.multi:146 [INFO] found job 16896450 in provider checkpoint proc_lsst.multi INFO: found job 16896450 in provider checkpoint 2024-03-07 15:15:21 proc_lsst.multi:201 [INFO] cancelling 16896450 on provider checkpoint proc_lsst.multi INFO: cancelling 16896450 on provider checkpoint 2024-03-07 15:15:21 proc_lsst.multi:146 [INFO] found job 16896423 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896423 in provider escience_rtx6k 2024-03-07 15:15:21 proc_lsst.multi:201 [INFO] cancelling 16896423 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16896423 on provider escience_rtx6k 2024-03-07 15:15:22 proc_lsst.multi:146 [INFO] found job 16896429 in provider escience_a40 proc_lsst.multi INFO: found job 16896429 in provider escience_a40 2024-03-07 15:15:22 proc_lsst.multi:201 [INFO] cancelling 16896429 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896429 on provider escience_a40 2024-03-07 15:15:22 proc_lsst.multi:146 [INFO] found job 16896389 in provider astro proc_lsst.multi INFO: found job 16896389 in provider astro 2024-03-07 15:15:22 proc_lsst.multi:201 [INFO] cancelling 16896389 on provider astro proc_lsst.multi INFO: cancelling 16896389 on provider astro 2024-03-07 15:15:22 proc_lsst.multi:146 [INFO] found job 16896422 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896422 in provider escience_rtx6k 2024-03-07 15:15:22 proc_lsst.multi:201 [INFO] cancelling 16896422 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16896422 on provider escience_rtx6k 2024-03-07 15:15:22 proc_lsst.multi:146 [INFO] found job 16896453 in provider checkpoint proc_lsst.multi INFO: found job 16896453 in provider checkpoint 2024-03-07 15:15:22 proc_lsst.multi:201 [INFO] cancelling 16896453 on provider checkpoint proc_lsst.multi INFO: cancelling 16896453 on provider checkpoint 2024-03-07 15:15:22 proc_lsst.multi:146 [INFO] found job 16896424 in provider escience_a40 proc_lsst.multi INFO: found job 16896424 in provider escience_a40 2024-03-07 15:15:22 proc_lsst.multi:201 [INFO] cancelling 16896424 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896424 on provider escience_a40 2024-03-07 15:15:22 proc_lsst.multi:146 [INFO] found job 16896431 in provider escience_a40 proc_lsst.multi INFO: found job 16896431 in provider escience_a40 2024-03-07 15:15:22 proc_lsst.multi:201 [INFO] cancelling 16896431 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896431 on provider escience_a40 2024-03-07 15:15:22 proc_lsst.multi:146 [INFO] found job 16896387 in provider astro proc_lsst.multi INFO: found job 16896387 in provider astro 2024-03-07 15:15:22 proc_lsst.multi:201 [INFO] cancelling 16896387 on provider astro proc_lsst.multi INFO: cancelling 16896387 on provider astro 2024-03-07 15:15:22 proc_lsst.multi:146 [INFO] found job 16896452 in provider checkpoint proc_lsst.multi INFO: found job 16896452 in provider checkpoint 2024-03-07 15:15:22 proc_lsst.multi:201 [INFO] cancelling 16896452 on provider checkpoint proc_lsst.multi INFO: cancelling 16896452 on provider checkpoint 2024-03-07 15:15:22 proc_lsst.multi:146 [INFO] found job 16896425 in provider escience_a40 proc_lsst.multi INFO: found job 16896425 in provider escience_a40 2024-03-07 15:15:22 proc_lsst.multi:201 [INFO] cancelling 16896425 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896425 on provider escience_a40 2024-03-07 15:15:23 proc_lsst.multi:146 [INFO] found job 16896421 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896421 in provider escience_rtx6k 2024-03-07 15:15:23 proc_lsst.multi:201 [INFO] cancelling 16896421 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16896421 on provider escience_rtx6k 2024-03-07 15:15:23 proc_lsst.multi:146 [INFO] found job 16896451 in provider checkpoint proc_lsst.multi INFO: found job 16896451 in provider checkpoint 2024-03-07 15:15:23 proc_lsst.multi:201 [INFO] cancelling 16896451 on provider checkpoint proc_lsst.multi INFO: cancelling 16896451 on provider checkpoint 2024-03-07 15:15:23 proc_lsst.multi:146 [INFO] found job 16896428 in provider escience_a40 proc_lsst.multi INFO: found job 16896428 in provider escience_a40 2024-03-07 15:15:23 proc_lsst.multi:201 [INFO] cancelling 16896428 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896428 on provider escience_a40 2024-03-07 15:15:23 proc_lsst.multi:146 [INFO] found job 16896394 in provider astro proc_lsst.multi INFO: found job 16896394 in provider astro 2024-03-07 15:15:23 proc_lsst.multi:201 [INFO] cancelling 16896394 on provider astro proc_lsst.multi INFO: cancelling 16896394 on provider astro 2024-03-07 15:15:23 proc_lsst.multi:146 [INFO] found job 16896427 in provider escience_a40 proc_lsst.multi INFO: found job 16896427 in provider escience_a40 2024-03-07 15:15:23 proc_lsst.multi:201 [INFO] cancelling 16896427 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896427 on provider escience_a40 2024-03-07 15:15:23 proc_lsst.multi:146 [INFO] found job 16896430 in provider escience_a40 proc_lsst.multi INFO: found job 16896430 in provider escience_a40 2024-03-07 15:15:23 proc_lsst.multi:201 [INFO] cancelling 16896430 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896430 on provider escience_a40 2024-03-07 15:15:23 proc_lsst.multi:146 [INFO] found job 16896420 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896420 in provider escience_rtx6k 2024-03-07 15:15:23 proc_lsst.multi:201 [INFO] cancelling 16896420 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16896420 on provider escience_rtx6k 2024-03-07 15:15:23 proc_lsst.multi:146 [INFO] found job 16896426 in provider escience_a40 proc_lsst.multi INFO: found job 16896426 in provider escience_a40 2024-03-07 15:15:23 proc_lsst.multi:201 [INFO] cancelling 16896426 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896426 on provider escience_a40 2024-03-07 15:15:23 proc_lsst.multi:146 [INFO] found job 16896386 in provider astro proc_lsst.multi INFO: found job 16896386 in provider astro 2024-03-07 15:15:23 proc_lsst.multi:201 [INFO] cancelling 16896386 on provider astro proc_lsst.multi INFO: cancelling 16896386 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-07 15:15:24 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-07 15:15:24 proc_lsst.multi:47 [INFO] new jobs since last cancel ['16896445', '16896446', '16896447', '16896448', '16896449', '16896450', '16896451', '16896452', '16896453', '16896454', '16896455', '16896456', '16896457', '16896458', '16896459', '16896460', '16896461', '16896462', '16896463', '16896464', '16896466', '16896467', '16896468', '16896469', '16896470', '16896471', '16896472', '16896473', '16896474', '16896475', '16896476', '16896477', '16896478', '16896479', '16896480', '16896481', '16896482', '16896483', '16896484', '16896485', '16896486', '16896487', '16896488', '16896489', '16896490', '16896491', '16896492', '16896493', '16896494', '16896495', '16896496', '16896497', '16896498', '16896499', '16896500', '16896501', '16896502', '16896503', '16896504', '16896505', '16896506', '16896507', '16896508', '16896509', '16896510', '16896511', '16896519', '16896520', '16896521', '16896522', '16896523', '16896524', '16896525', '16896526', '16896527', '16896528', '16896529', '16896530', '16896531', '16896533', '16896534', '16896535', '16896536', '16896537', '16896538', '16896539', '16896540', '16896541', '16896542', '16896543', '16896544', '16896545', '16896546', '16896547', '16896548', '16896549', '16896550', '16896551', '16896552', '16896553', '16896554', '16896555', '16896556', '16896559', '16896561', '16896562', '16896563', '16896564', '16896565', '16896566', '16896568', '16896569', '16896570', '16896571', '16896572', '16896573', '16896574', '16896575', '16896576', '16896577', '16896578', '16896579', '16896580', '16896581', '16896582', '16896583', '16896584', '16896585', '16896586', '16896587', '16896588', '16896589', '16896590', '16896591', '16896592', '16896593', '16896594', '16896595', '16896596', '16896597', '16896598', '16896599', '16896600', '16896601', '16896602', '16896603', '16896604', '16896605', '16896606', '16896607', '16896608', '16896609', '16896610', '16896611', '16896612', '16896613', '16896614', '16896615', '16896616', '16896617', '16896618', '16896619', '16896620', '16896621', '16896622', '16896623', '16896624', '16896625', '16896626', '16896627', '16896628', '16896629', '16896630', '16896631', '16896632', '16896633', '16896634', '16896635', '16896636', '16896637', '16896638', '16896639', '16896640', '16896641', '16896642', '16896643', '16896644', '16896645', '16896646', '16896647', '16896648', '16896649', '16896650', '16896651', '16896652', '16896653', '16896654', '16896655', '16896656', '16896657', '16896658', '16896659', '16896660', '16896661', '16896662', '16896663', '16896664', '16896665', '16896666', '16896667', '16896668', '16896669', '16896670', '16896671', '16896672', '16896673', '16896674', '16896675', '16896676', '16896678', '16896679', '16896680', '16896681', '16896682', '16896683', '16896684', '16896685', '16896686', '16896687', '16896688', '16896689', '16896690', '16896691', '16896692', '16896693', '16896694', '16896695', '16896696', '16896697', '16896698', '16896699', '16896700', '16896701', '16896702', '16896703', '16896704', '16896705', '16896706', '16896707', '16896708', '16896709', '16896710', '16896711', '16896712', '16896713', '16896714', '16896385', '16896386', '16896387', '16896388', '16896389', '16896390', '16896392', '16896394', '16896395', '16896396', '16896397', '16896398', '16896399', '16896400', '16896401', '16896402', '16896403', '16896404', '16896405', '16896408', '16896409', '16896410', '16896411', '16896412', '16896413', '16896414', '16896415', '16896416', '16896417', '16896418', '16896424', '16896425', '16896426', '16896427', '16896428', '16896429', '16896430', '16896431', '16896432', '16896433', '16896434', '16896435', '16896437', '16896438', '16896439', '16896440', '16896441', '16896442', '16896443', '16896444', '16896419', '16896420', '16896421', '16896422', '16896423', '53087'] proc_lsst.multi INFO: new jobs since last cancel ['16896445', '16896446', '16896447', '16896448', '16896449', '16896450', '16896451', '16896452', '16896453', '16896454', '16896455', '16896456', '16896457', '16896458', '16896459', '16896460', '16896461', '16896462', '16896463', '16896464', '16896466', '16896467', '16896468', '16896469', '16896470', '16896471', '16896472', '16896473', '16896474', '16896475', '16896476', '16896477', '16896478', '16896479', '16896480', '16896481', '16896482', '16896483', '16896484', '16896485', '16896486', '16896487', '16896488', '16896489', '16896490', '16896491', '16896492', '16896493', '16896494', '16896495', '16896496', '16896497', '16896498', '16896499', '16896500', '16896501', '16896502', '16896503', '16896504', '16896505', '16896506', '16896507', '16896508', '16896509', '16896510', '16896511', '16896519', '16896520', '16896521', '16896522', '16896523', '16896524', '16896525', '16896526', '16896527', '16896528', '16896529', '16896530', '16896531', '16896533', '16896534', '16896535', '16896536', '16896537', '16896538', '16896539', '16896540', '16896541', '16896542', '16896543', '16896544', '16896545', '16896546', '16896547', '16896548', '16896549', '16896550', '16896551', '16896552', '16896553', '16896554', '16896555', '16896556', '16896559', '16896561', '16896562', '16896563', '16896564', '16896565', '16896566', '16896568', '16896569', '16896570', '16896571', '16896572', '16896573', '16896574', '16896575', '16896576', '16896577', '16896578', '16896579', '16896580', '16896581', '16896582', '16896583', '16896584', '16896585', '16896586', '16896587', '16896588', '16896589', '16896590', '16896591', '16896592', '16896593', '16896594', '16896595', '16896596', '16896597', '16896598', '16896599', '16896600', '16896601', '16896602', '16896603', '16896604', '16896605', '16896606', '16896607', '16896608', '16896609', '16896610', '16896611', '16896612', '16896613', '16896614', '16896615', '16896616', '16896617', '16896618', '16896619', '16896620', '16896621', '16896622', '16896623', '16896624', '16896625', '16896626', '16896627', '16896628', '16896629', '16896630', '16896631', '16896632', '16896633', '16896634', '16896635', '16896636', '16896637', '16896638', '16896639', '16896640', '16896641', '16896642', '16896643', '16896644', '16896645', '16896646', '16896647', '16896648', '16896649', '16896650', '16896651', '16896652', '16896653', '16896654', '16896655', '16896656', '16896657', '16896658', '16896659', '16896660', '16896661', '16896662', '16896663', '16896664', '16896665', '16896666', '16896667', '16896668', '16896669', '16896670', '16896671', '16896672', '16896673', '16896674', '16896675', '16896676', '16896678', '16896679', '16896680', '16896681', '16896682', '16896683', '16896684', '16896685', '16896686', '16896687', '16896688', '16896689', '16896690', '16896691', '16896692', '16896693', '16896694', '16896695', '16896696', '16896697', '16896698', '16896699', '16896700', '16896701', '16896702', '16896703', '16896704', '16896705', '16896706', '16896707', '16896708', '16896709', '16896710', '16896711', '16896712', '16896713', '16896714', '16896385', '16896386', '16896387', '16896388', '16896389', '16896390', '16896392', '16896394', '16896395', '16896396', '16896397', '16896398', '16896399', '16896400', '16896401', '16896402', '16896403', '16896404', '16896405', '16896408', '16896409', '16896410', '16896411', '16896412', '16896413', '16896414', '16896415', '16896416', '16896417', '16896418', '16896424', '16896425', '16896426', '16896427', '16896428', '16896429', '16896430', '16896431', '16896432', '16896433', '16896434', '16896435', '16896437', '16896438', '16896439', '16896440', '16896441', '16896442', '16896443', '16896444', '16896419', '16896420', '16896421', '16896422', '16896423', '53087'] 2024-03-07 15:15:24 proc_lsst.multi:146 [INFO] found job 16896445 in provider checkpoint proc_lsst.multi INFO: found job 16896445 in provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:201 [INFO] cancelling 16896445 on provider checkpoint proc_lsst.multi INFO: cancelling 16896445 on provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:146 [INFO] found job 16896446 in provider checkpoint proc_lsst.multi INFO: found job 16896446 in provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:201 [INFO] cancelling 16896446 on provider checkpoint proc_lsst.multi INFO: cancelling 16896446 on provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:146 [INFO] found job 16896447 in provider checkpoint proc_lsst.multi INFO: found job 16896447 in provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:201 [INFO] cancelling 16896447 on provider checkpoint proc_lsst.multi INFO: cancelling 16896447 on provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:146 [INFO] found job 16896448 in provider checkpoint proc_lsst.multi INFO: found job 16896448 in provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:201 [INFO] cancelling 16896448 on provider checkpoint proc_lsst.multi INFO: cancelling 16896448 on provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:146 [INFO] found job 16896449 in provider checkpoint proc_lsst.multi INFO: found job 16896449 in provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:201 [INFO] cancelling 16896449 on provider checkpoint proc_lsst.multi INFO: cancelling 16896449 on provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:146 [INFO] found job 16896450 in provider checkpoint proc_lsst.multi INFO: found job 16896450 in provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:201 [INFO] cancelling 16896450 on provider checkpoint proc_lsst.multi INFO: cancelling 16896450 on provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:146 [INFO] found job 16896451 in provider checkpoint proc_lsst.multi INFO: found job 16896451 in provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:201 [INFO] cancelling 16896451 on provider checkpoint proc_lsst.multi INFO: cancelling 16896451 on provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:146 [INFO] found job 16896452 in provider checkpoint proc_lsst.multi INFO: found job 16896452 in provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:201 [INFO] cancelling 16896452 on provider checkpoint proc_lsst.multi INFO: cancelling 16896452 on provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:146 [INFO] found job 16896453 in provider checkpoint proc_lsst.multi INFO: found job 16896453 in provider checkpoint 2024-03-07 15:15:24 proc_lsst.multi:201 [INFO] cancelling 16896453 on provider checkpoint proc_lsst.multi INFO: cancelling 16896453 on provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:146 [INFO] found job 16896454 in provider checkpoint proc_lsst.multi INFO: found job 16896454 in provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:201 [INFO] cancelling 16896454 on provider checkpoint proc_lsst.multi INFO: cancelling 16896454 on provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:146 [INFO] found job 16896455 in provider checkpoint proc_lsst.multi INFO: found job 16896455 in provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:201 [INFO] cancelling 16896455 on provider checkpoint proc_lsst.multi INFO: cancelling 16896455 on provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:146 [INFO] found job 16896456 in provider checkpoint proc_lsst.multi INFO: found job 16896456 in provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:201 [INFO] cancelling 16896456 on provider checkpoint proc_lsst.multi INFO: cancelling 16896456 on provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:146 [INFO] found job 16896457 in provider checkpoint proc_lsst.multi INFO: found job 16896457 in provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:201 [INFO] cancelling 16896457 on provider checkpoint proc_lsst.multi INFO: cancelling 16896457 on provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:146 [INFO] found job 16896458 in provider checkpoint proc_lsst.multi INFO: found job 16896458 in provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:201 [INFO] cancelling 16896458 on provider checkpoint proc_lsst.multi INFO: cancelling 16896458 on provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:146 [INFO] found job 16896459 in provider checkpoint proc_lsst.multi INFO: found job 16896459 in provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:201 [INFO] cancelling 16896459 on provider checkpoint proc_lsst.multi INFO: cancelling 16896459 on provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:146 [INFO] found job 16896460 in provider checkpoint proc_lsst.multi INFO: found job 16896460 in provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:201 [INFO] cancelling 16896460 on provider checkpoint proc_lsst.multi INFO: cancelling 16896460 on provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:146 [INFO] found job 16896461 in provider checkpoint proc_lsst.multi INFO: found job 16896461 in provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:201 [INFO] cancelling 16896461 on provider checkpoint proc_lsst.multi INFO: cancelling 16896461 on provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:146 [INFO] found job 16896462 in provider checkpoint proc_lsst.multi INFO: found job 16896462 in provider checkpoint 2024-03-07 15:15:25 proc_lsst.multi:201 [INFO] cancelling 16896462 on provider checkpoint proc_lsst.multi INFO: cancelling 16896462 on provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:146 [INFO] found job 16896463 in provider checkpoint proc_lsst.multi INFO: found job 16896463 in provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:201 [INFO] cancelling 16896463 on provider checkpoint proc_lsst.multi INFO: cancelling 16896463 on provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:146 [INFO] found job 16896464 in provider checkpoint proc_lsst.multi INFO: found job 16896464 in provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:201 [INFO] cancelling 16896464 on provider checkpoint proc_lsst.multi INFO: cancelling 16896464 on provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:146 [INFO] found job 16896466 in provider checkpoint proc_lsst.multi INFO: found job 16896466 in provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:201 [INFO] cancelling 16896466 on provider checkpoint proc_lsst.multi INFO: cancelling 16896466 on provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:146 [INFO] found job 16896467 in provider checkpoint proc_lsst.multi INFO: found job 16896467 in provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:201 [INFO] cancelling 16896467 on provider checkpoint proc_lsst.multi INFO: cancelling 16896467 on provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:146 [INFO] found job 16896468 in provider checkpoint proc_lsst.multi INFO: found job 16896468 in provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:201 [INFO] cancelling 16896468 on provider checkpoint proc_lsst.multi INFO: cancelling 16896468 on provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:146 [INFO] found job 16896469 in provider checkpoint proc_lsst.multi INFO: found job 16896469 in provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:201 [INFO] cancelling 16896469 on provider checkpoint proc_lsst.multi INFO: cancelling 16896469 on provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:146 [INFO] found job 16896470 in provider checkpoint proc_lsst.multi INFO: found job 16896470 in provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:201 [INFO] cancelling 16896470 on provider checkpoint proc_lsst.multi INFO: cancelling 16896470 on provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:146 [INFO] found job 16896471 in provider checkpoint proc_lsst.multi INFO: found job 16896471 in provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:201 [INFO] cancelling 16896471 on provider checkpoint proc_lsst.multi INFO: cancelling 16896471 on provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:146 [INFO] found job 16896472 in provider checkpoint proc_lsst.multi INFO: found job 16896472 in provider checkpoint 2024-03-07 15:15:26 proc_lsst.multi:201 [INFO] cancelling 16896472 on provider checkpoint proc_lsst.multi INFO: cancelling 16896472 on provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:146 [INFO] found job 16896473 in provider checkpoint proc_lsst.multi INFO: found job 16896473 in provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:201 [INFO] cancelling 16896473 on provider checkpoint proc_lsst.multi INFO: cancelling 16896473 on provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:146 [INFO] found job 16896474 in provider checkpoint proc_lsst.multi INFO: found job 16896474 in provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:201 [INFO] cancelling 16896474 on provider checkpoint proc_lsst.multi INFO: cancelling 16896474 on provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:146 [INFO] found job 16896475 in provider checkpoint proc_lsst.multi INFO: found job 16896475 in provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:201 [INFO] cancelling 16896475 on provider checkpoint proc_lsst.multi INFO: cancelling 16896475 on provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:146 [INFO] found job 16896476 in provider checkpoint proc_lsst.multi INFO: found job 16896476 in provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:201 [INFO] cancelling 16896476 on provider checkpoint proc_lsst.multi INFO: cancelling 16896476 on provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:146 [INFO] found job 16896477 in provider checkpoint proc_lsst.multi INFO: found job 16896477 in provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:201 [INFO] cancelling 16896477 on provider checkpoint proc_lsst.multi INFO: cancelling 16896477 on provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:146 [INFO] found job 16896478 in provider checkpoint proc_lsst.multi INFO: found job 16896478 in provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:201 [INFO] cancelling 16896478 on provider checkpoint proc_lsst.multi INFO: cancelling 16896478 on provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:146 [INFO] found job 16896479 in provider checkpoint proc_lsst.multi INFO: found job 16896479 in provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:201 [INFO] cancelling 16896479 on provider checkpoint proc_lsst.multi INFO: cancelling 16896479 on provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:146 [INFO] found job 16896480 in provider checkpoint proc_lsst.multi INFO: found job 16896480 in provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:201 [INFO] cancelling 16896480 on provider checkpoint proc_lsst.multi INFO: cancelling 16896480 on provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:146 [INFO] found job 16896481 in provider checkpoint proc_lsst.multi INFO: found job 16896481 in provider checkpoint 2024-03-07 15:15:27 proc_lsst.multi:201 [INFO] cancelling 16896481 on provider checkpoint proc_lsst.multi INFO: cancelling 16896481 on provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:146 [INFO] found job 16896482 in provider checkpoint proc_lsst.multi INFO: found job 16896482 in provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:201 [INFO] cancelling 16896482 on provider checkpoint proc_lsst.multi INFO: cancelling 16896482 on provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:146 [INFO] found job 16896483 in provider checkpoint proc_lsst.multi INFO: found job 16896483 in provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:201 [INFO] cancelling 16896483 on provider checkpoint proc_lsst.multi INFO: cancelling 16896483 on provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:146 [INFO] found job 16896484 in provider checkpoint proc_lsst.multi INFO: found job 16896484 in provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:201 [INFO] cancelling 16896484 on provider checkpoint proc_lsst.multi INFO: cancelling 16896484 on provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:146 [INFO] found job 16896485 in provider checkpoint proc_lsst.multi INFO: found job 16896485 in provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:201 [INFO] cancelling 16896485 on provider checkpoint proc_lsst.multi INFO: cancelling 16896485 on provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:146 [INFO] found job 16896486 in provider checkpoint proc_lsst.multi INFO: found job 16896486 in provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:201 [INFO] cancelling 16896486 on provider checkpoint proc_lsst.multi INFO: cancelling 16896486 on provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:146 [INFO] found job 16896487 in provider checkpoint proc_lsst.multi INFO: found job 16896487 in provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:201 [INFO] cancelling 16896487 on provider checkpoint proc_lsst.multi INFO: cancelling 16896487 on provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:146 [INFO] found job 16896488 in provider checkpoint proc_lsst.multi INFO: found job 16896488 in provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:201 [INFO] cancelling 16896488 on provider checkpoint proc_lsst.multi INFO: cancelling 16896488 on provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:146 [INFO] found job 16896489 in provider checkpoint proc_lsst.multi INFO: found job 16896489 in provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:201 [INFO] cancelling 16896489 on provider checkpoint proc_lsst.multi INFO: cancelling 16896489 on provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:146 [INFO] found job 16896490 in provider checkpoint proc_lsst.multi INFO: found job 16896490 in provider checkpoint 2024-03-07 15:15:28 proc_lsst.multi:201 [INFO] cancelling 16896490 on provider checkpoint proc_lsst.multi INFO: cancelling 16896490 on provider checkpoint 2024-03-07 15:15:29 proc_lsst.multi:146 [INFO] found job 16896491 in provider checkpoint proc_lsst.multi INFO: found job 16896491 in provider checkpoint 2024-03-07 15:15:29 proc_lsst.multi:201 [INFO] cancelling 16896491 on provider checkpoint proc_lsst.multi INFO: cancelling 16896491 on provider checkpoint 2024-03-07 15:15:29 proc_lsst.multi:146 [INFO] found job 16896492 in provider checkpoint proc_lsst.multi INFO: found job 16896492 in provider checkpoint 2024-03-07 15:15:29 proc_lsst.multi:201 [INFO] cancelling 16896492 on provider checkpoint proc_lsst.multi INFO: cancelling 16896492 on provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:146 [INFO] found job 16896493 in provider checkpoint proc_lsst.multi INFO: found job 16896493 in provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:201 [INFO] cancelling 16896493 on provider checkpoint proc_lsst.multi INFO: cancelling 16896493 on provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:146 [INFO] found job 16896494 in provider checkpoint proc_lsst.multi INFO: found job 16896494 in provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:201 [INFO] cancelling 16896494 on provider checkpoint proc_lsst.multi INFO: cancelling 16896494 on provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:146 [INFO] found job 16896495 in provider checkpoint proc_lsst.multi INFO: found job 16896495 in provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:201 [INFO] cancelling 16896495 on provider checkpoint proc_lsst.multi INFO: cancelling 16896495 on provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:146 [INFO] found job 16896496 in provider checkpoint proc_lsst.multi INFO: found job 16896496 in provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:201 [INFO] cancelling 16896496 on provider checkpoint proc_lsst.multi INFO: cancelling 16896496 on provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:146 [INFO] found job 16896497 in provider checkpoint proc_lsst.multi INFO: found job 16896497 in provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:201 [INFO] cancelling 16896497 on provider checkpoint proc_lsst.multi INFO: cancelling 16896497 on provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:146 [INFO] found job 16896498 in provider checkpoint proc_lsst.multi INFO: found job 16896498 in provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:201 [INFO] cancelling 16896498 on provider checkpoint proc_lsst.multi INFO: cancelling 16896498 on provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:146 [INFO] found job 16896499 in provider checkpoint proc_lsst.multi INFO: found job 16896499 in provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:201 [INFO] cancelling 16896499 on provider checkpoint proc_lsst.multi INFO: cancelling 16896499 on provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:146 [INFO] found job 16896500 in provider checkpoint proc_lsst.multi INFO: found job 16896500 in provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:201 [INFO] cancelling 16896500 on provider checkpoint proc_lsst.multi INFO: cancelling 16896500 on provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:146 [INFO] found job 16896501 in provider checkpoint proc_lsst.multi INFO: found job 16896501 in provider checkpoint 2024-03-07 15:15:31 proc_lsst.multi:201 [INFO] cancelling 16896501 on provider checkpoint proc_lsst.multi INFO: cancelling 16896501 on provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:146 [INFO] found job 16896502 in provider checkpoint proc_lsst.multi INFO: found job 16896502 in provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:201 [INFO] cancelling 16896502 on provider checkpoint proc_lsst.multi INFO: cancelling 16896502 on provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:146 [INFO] found job 16896503 in provider checkpoint proc_lsst.multi INFO: found job 16896503 in provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:201 [INFO] cancelling 16896503 on provider checkpoint proc_lsst.multi INFO: cancelling 16896503 on provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:146 [INFO] found job 16896504 in provider checkpoint proc_lsst.multi INFO: found job 16896504 in provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:201 [INFO] cancelling 16896504 on provider checkpoint proc_lsst.multi INFO: cancelling 16896504 on provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:146 [INFO] found job 16896505 in provider checkpoint proc_lsst.multi INFO: found job 16896505 in provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:201 [INFO] cancelling 16896505 on provider checkpoint proc_lsst.multi INFO: cancelling 16896505 on provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:146 [INFO] found job 16896506 in provider checkpoint proc_lsst.multi INFO: found job 16896506 in provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:201 [INFO] cancelling 16896506 on provider checkpoint proc_lsst.multi INFO: cancelling 16896506 on provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:146 [INFO] found job 16896507 in provider checkpoint proc_lsst.multi INFO: found job 16896507 in provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:201 [INFO] cancelling 16896507 on provider checkpoint proc_lsst.multi INFO: cancelling 16896507 on provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:146 [INFO] found job 16896508 in provider checkpoint proc_lsst.multi INFO: found job 16896508 in provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:201 [INFO] cancelling 16896508 on provider checkpoint proc_lsst.multi INFO: cancelling 16896508 on provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:146 [INFO] found job 16896509 in provider checkpoint proc_lsst.multi INFO: found job 16896509 in provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:201 [INFO] cancelling 16896509 on provider checkpoint proc_lsst.multi INFO: cancelling 16896509 on provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:146 [INFO] found job 16896510 in provider checkpoint proc_lsst.multi INFO: found job 16896510 in provider checkpoint 2024-03-07 15:15:32 proc_lsst.multi:201 [INFO] cancelling 16896510 on provider checkpoint proc_lsst.multi INFO: cancelling 16896510 on provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:146 [INFO] found job 16896511 in provider checkpoint proc_lsst.multi INFO: found job 16896511 in provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:201 [INFO] cancelling 16896511 on provider checkpoint proc_lsst.multi INFO: cancelling 16896511 on provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:146 [INFO] found job 16896519 in provider checkpoint proc_lsst.multi INFO: found job 16896519 in provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:201 [INFO] cancelling 16896519 on provider checkpoint proc_lsst.multi INFO: cancelling 16896519 on provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:146 [INFO] found job 16896520 in provider checkpoint proc_lsst.multi INFO: found job 16896520 in provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:201 [INFO] cancelling 16896520 on provider checkpoint proc_lsst.multi INFO: cancelling 16896520 on provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:146 [INFO] found job 16896521 in provider checkpoint proc_lsst.multi INFO: found job 16896521 in provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:201 [INFO] cancelling 16896521 on provider checkpoint proc_lsst.multi INFO: cancelling 16896521 on provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:146 [INFO] found job 16896522 in provider checkpoint proc_lsst.multi INFO: found job 16896522 in provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:201 [INFO] cancelling 16896522 on provider checkpoint proc_lsst.multi INFO: cancelling 16896522 on provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:146 [INFO] found job 16896523 in provider checkpoint proc_lsst.multi INFO: found job 16896523 in provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:201 [INFO] cancelling 16896523 on provider checkpoint proc_lsst.multi INFO: cancelling 16896523 on provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:146 [INFO] found job 16896524 in provider checkpoint proc_lsst.multi INFO: found job 16896524 in provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:201 [INFO] cancelling 16896524 on provider checkpoint proc_lsst.multi INFO: cancelling 16896524 on provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:146 [INFO] found job 16896525 in provider checkpoint proc_lsst.multi INFO: found job 16896525 in provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:201 [INFO] cancelling 16896525 on provider checkpoint proc_lsst.multi INFO: cancelling 16896525 on provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:146 [INFO] found job 16896526 in provider checkpoint proc_lsst.multi INFO: found job 16896526 in provider checkpoint 2024-03-07 15:15:33 proc_lsst.multi:201 [INFO] cancelling 16896526 on provider checkpoint proc_lsst.multi INFO: cancelling 16896526 on provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:146 [INFO] found job 16896527 in provider checkpoint proc_lsst.multi INFO: found job 16896527 in provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:201 [INFO] cancelling 16896527 on provider checkpoint proc_lsst.multi INFO: cancelling 16896527 on provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:146 [INFO] found job 16896528 in provider checkpoint proc_lsst.multi INFO: found job 16896528 in provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:201 [INFO] cancelling 16896528 on provider checkpoint proc_lsst.multi INFO: cancelling 16896528 on provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:146 [INFO] found job 16896529 in provider checkpoint proc_lsst.multi INFO: found job 16896529 in provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:201 [INFO] cancelling 16896529 on provider checkpoint proc_lsst.multi INFO: cancelling 16896529 on provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:146 [INFO] found job 16896530 in provider checkpoint proc_lsst.multi INFO: found job 16896530 in provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:201 [INFO] cancelling 16896530 on provider checkpoint proc_lsst.multi INFO: cancelling 16896530 on provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:146 [INFO] found job 16896531 in provider checkpoint proc_lsst.multi INFO: found job 16896531 in provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:201 [INFO] cancelling 16896531 on provider checkpoint proc_lsst.multi INFO: cancelling 16896531 on provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:146 [INFO] found job 16896533 in provider checkpoint proc_lsst.multi INFO: found job 16896533 in provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:201 [INFO] cancelling 16896533 on provider checkpoint proc_lsst.multi INFO: cancelling 16896533 on provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:146 [INFO] found job 16896534 in provider checkpoint proc_lsst.multi INFO: found job 16896534 in provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:201 [INFO] cancelling 16896534 on provider checkpoint proc_lsst.multi INFO: cancelling 16896534 on provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:146 [INFO] found job 16896535 in provider checkpoint proc_lsst.multi INFO: found job 16896535 in provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:201 [INFO] cancelling 16896535 on provider checkpoint proc_lsst.multi INFO: cancelling 16896535 on provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:146 [INFO] found job 16896536 in provider checkpoint proc_lsst.multi INFO: found job 16896536 in provider checkpoint 2024-03-07 15:15:34 proc_lsst.multi:201 [INFO] cancelling 16896536 on provider checkpoint proc_lsst.multi INFO: cancelling 16896536 on provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:146 [INFO] found job 16896537 in provider checkpoint proc_lsst.multi INFO: found job 16896537 in provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:201 [INFO] cancelling 16896537 on provider checkpoint proc_lsst.multi INFO: cancelling 16896537 on provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:146 [INFO] found job 16896538 in provider checkpoint proc_lsst.multi INFO: found job 16896538 in provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:201 [INFO] cancelling 16896538 on provider checkpoint proc_lsst.multi INFO: cancelling 16896538 on provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:146 [INFO] found job 16896539 in provider checkpoint proc_lsst.multi INFO: found job 16896539 in provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:201 [INFO] cancelling 16896539 on provider checkpoint proc_lsst.multi INFO: cancelling 16896539 on provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:146 [INFO] found job 16896540 in provider checkpoint proc_lsst.multi INFO: found job 16896540 in provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:201 [INFO] cancelling 16896540 on provider checkpoint proc_lsst.multi INFO: cancelling 16896540 on provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:146 [INFO] found job 16896541 in provider checkpoint proc_lsst.multi INFO: found job 16896541 in provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:201 [INFO] cancelling 16896541 on provider checkpoint proc_lsst.multi INFO: cancelling 16896541 on provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:146 [INFO] found job 16896542 in provider checkpoint proc_lsst.multi INFO: found job 16896542 in provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:201 [INFO] cancelling 16896542 on provider checkpoint proc_lsst.multi INFO: cancelling 16896542 on provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:146 [INFO] found job 16896543 in provider checkpoint proc_lsst.multi INFO: found job 16896543 in provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:201 [INFO] cancelling 16896543 on provider checkpoint proc_lsst.multi INFO: cancelling 16896543 on provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:146 [INFO] found job 16896544 in provider checkpoint proc_lsst.multi INFO: found job 16896544 in provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:201 [INFO] cancelling 16896544 on provider checkpoint proc_lsst.multi INFO: cancelling 16896544 on provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:146 [INFO] found job 16896545 in provider checkpoint proc_lsst.multi INFO: found job 16896545 in provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:201 [INFO] cancelling 16896545 on provider checkpoint proc_lsst.multi INFO: cancelling 16896545 on provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:146 [INFO] found job 16896546 in provider checkpoint proc_lsst.multi INFO: found job 16896546 in provider checkpoint 2024-03-07 15:15:35 proc_lsst.multi:201 [INFO] cancelling 16896546 on provider checkpoint proc_lsst.multi INFO: cancelling 16896546 on provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:146 [INFO] found job 16896547 in provider checkpoint proc_lsst.multi INFO: found job 16896547 in provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:201 [INFO] cancelling 16896547 on provider checkpoint proc_lsst.multi INFO: cancelling 16896547 on provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:146 [INFO] found job 16896548 in provider checkpoint proc_lsst.multi INFO: found job 16896548 in provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:201 [INFO] cancelling 16896548 on provider checkpoint proc_lsst.multi INFO: cancelling 16896548 on provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:146 [INFO] found job 16896549 in provider checkpoint proc_lsst.multi INFO: found job 16896549 in provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:201 [INFO] cancelling 16896549 on provider checkpoint proc_lsst.multi INFO: cancelling 16896549 on provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:146 [INFO] found job 16896550 in provider checkpoint proc_lsst.multi INFO: found job 16896550 in provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:201 [INFO] cancelling 16896550 on provider checkpoint proc_lsst.multi INFO: cancelling 16896550 on provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:146 [INFO] found job 16896551 in provider checkpoint proc_lsst.multi INFO: found job 16896551 in provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:201 [INFO] cancelling 16896551 on provider checkpoint proc_lsst.multi INFO: cancelling 16896551 on provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:146 [INFO] found job 16896552 in provider checkpoint proc_lsst.multi INFO: found job 16896552 in provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:201 [INFO] cancelling 16896552 on provider checkpoint proc_lsst.multi INFO: cancelling 16896552 on provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:146 [INFO] found job 16896553 in provider checkpoint proc_lsst.multi INFO: found job 16896553 in provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:201 [INFO] cancelling 16896553 on provider checkpoint proc_lsst.multi INFO: cancelling 16896553 on provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:146 [INFO] found job 16896554 in provider checkpoint proc_lsst.multi INFO: found job 16896554 in provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:201 [INFO] cancelling 16896554 on provider checkpoint proc_lsst.multi INFO: cancelling 16896554 on provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:146 [INFO] found job 16896555 in provider checkpoint proc_lsst.multi INFO: found job 16896555 in provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:201 [INFO] cancelling 16896555 on provider checkpoint proc_lsst.multi INFO: cancelling 16896555 on provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:146 [INFO] found job 16896556 in provider checkpoint proc_lsst.multi INFO: found job 16896556 in provider checkpoint 2024-03-07 15:15:36 proc_lsst.multi:201 [INFO] cancelling 16896556 on provider checkpoint proc_lsst.multi INFO: cancelling 16896556 on provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:146 [INFO] found job 16896559 in provider checkpoint proc_lsst.multi INFO: found job 16896559 in provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:201 [INFO] cancelling 16896559 on provider checkpoint proc_lsst.multi INFO: cancelling 16896559 on provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:146 [INFO] found job 16896561 in provider checkpoint proc_lsst.multi INFO: found job 16896561 in provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:201 [INFO] cancelling 16896561 on provider checkpoint proc_lsst.multi INFO: cancelling 16896561 on provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:146 [INFO] found job 16896562 in provider checkpoint proc_lsst.multi INFO: found job 16896562 in provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:201 [INFO] cancelling 16896562 on provider checkpoint proc_lsst.multi INFO: cancelling 16896562 on provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:146 [INFO] found job 16896563 in provider checkpoint proc_lsst.multi INFO: found job 16896563 in provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:201 [INFO] cancelling 16896563 on provider checkpoint proc_lsst.multi INFO: cancelling 16896563 on provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:146 [INFO] found job 16896564 in provider checkpoint proc_lsst.multi INFO: found job 16896564 in provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:201 [INFO] cancelling 16896564 on provider checkpoint proc_lsst.multi INFO: cancelling 16896564 on provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:146 [INFO] found job 16896565 in provider checkpoint proc_lsst.multi INFO: found job 16896565 in provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:201 [INFO] cancelling 16896565 on provider checkpoint proc_lsst.multi INFO: cancelling 16896565 on provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:146 [INFO] found job 16896566 in provider checkpoint proc_lsst.multi INFO: found job 16896566 in provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:201 [INFO] cancelling 16896566 on provider checkpoint proc_lsst.multi INFO: cancelling 16896566 on provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:146 [INFO] found job 16896568 in provider checkpoint proc_lsst.multi INFO: found job 16896568 in provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:201 [INFO] cancelling 16896568 on provider checkpoint proc_lsst.multi INFO: cancelling 16896568 on provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:146 [INFO] found job 16896569 in provider checkpoint proc_lsst.multi INFO: found job 16896569 in provider checkpoint 2024-03-07 15:15:37 proc_lsst.multi:201 [INFO] cancelling 16896569 on provider checkpoint proc_lsst.multi INFO: cancelling 16896569 on provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:146 [INFO] found job 16896570 in provider checkpoint proc_lsst.multi INFO: found job 16896570 in provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:201 [INFO] cancelling 16896570 on provider checkpoint proc_lsst.multi INFO: cancelling 16896570 on provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:146 [INFO] found job 16896571 in provider checkpoint proc_lsst.multi INFO: found job 16896571 in provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:201 [INFO] cancelling 16896571 on provider checkpoint proc_lsst.multi INFO: cancelling 16896571 on provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:146 [INFO] found job 16896572 in provider checkpoint proc_lsst.multi INFO: found job 16896572 in provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:201 [INFO] cancelling 16896572 on provider checkpoint proc_lsst.multi INFO: cancelling 16896572 on provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:146 [INFO] found job 16896573 in provider checkpoint proc_lsst.multi INFO: found job 16896573 in provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:201 [INFO] cancelling 16896573 on provider checkpoint proc_lsst.multi INFO: cancelling 16896573 on provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:146 [INFO] found job 16896574 in provider checkpoint proc_lsst.multi INFO: found job 16896574 in provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:201 [INFO] cancelling 16896574 on provider checkpoint proc_lsst.multi INFO: cancelling 16896574 on provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:146 [INFO] found job 16896575 in provider checkpoint proc_lsst.multi INFO: found job 16896575 in provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:201 [INFO] cancelling 16896575 on provider checkpoint proc_lsst.multi INFO: cancelling 16896575 on provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:146 [INFO] found job 16896576 in provider checkpoint proc_lsst.multi INFO: found job 16896576 in provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:201 [INFO] cancelling 16896576 on provider checkpoint proc_lsst.multi INFO: cancelling 16896576 on provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:146 [INFO] found job 16896577 in provider checkpoint proc_lsst.multi INFO: found job 16896577 in provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:201 [INFO] cancelling 16896577 on provider checkpoint proc_lsst.multi INFO: cancelling 16896577 on provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:146 [INFO] found job 16896578 in provider checkpoint proc_lsst.multi INFO: found job 16896578 in provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:201 [INFO] cancelling 16896578 on provider checkpoint proc_lsst.multi INFO: cancelling 16896578 on provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:146 [INFO] found job 16896579 in provider checkpoint proc_lsst.multi INFO: found job 16896579 in provider checkpoint 2024-03-07 15:15:38 proc_lsst.multi:201 [INFO] cancelling 16896579 on provider checkpoint proc_lsst.multi INFO: cancelling 16896579 on provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:146 [INFO] found job 16896580 in provider checkpoint proc_lsst.multi INFO: found job 16896580 in provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:201 [INFO] cancelling 16896580 on provider checkpoint proc_lsst.multi INFO: cancelling 16896580 on provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:146 [INFO] found job 16896581 in provider checkpoint proc_lsst.multi INFO: found job 16896581 in provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:201 [INFO] cancelling 16896581 on provider checkpoint proc_lsst.multi INFO: cancelling 16896581 on provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:146 [INFO] found job 16896582 in provider checkpoint proc_lsst.multi INFO: found job 16896582 in provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:201 [INFO] cancelling 16896582 on provider checkpoint proc_lsst.multi INFO: cancelling 16896582 on provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:146 [INFO] found job 16896583 in provider checkpoint proc_lsst.multi INFO: found job 16896583 in provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:201 [INFO] cancelling 16896583 on provider checkpoint proc_lsst.multi INFO: cancelling 16896583 on provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:146 [INFO] found job 16896584 in provider checkpoint proc_lsst.multi INFO: found job 16896584 in provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:201 [INFO] cancelling 16896584 on provider checkpoint proc_lsst.multi INFO: cancelling 16896584 on provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:146 [INFO] found job 16896585 in provider checkpoint proc_lsst.multi INFO: found job 16896585 in provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:201 [INFO] cancelling 16896585 on provider checkpoint proc_lsst.multi INFO: cancelling 16896585 on provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:146 [INFO] found job 16896586 in provider checkpoint proc_lsst.multi INFO: found job 16896586 in provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:201 [INFO] cancelling 16896586 on provider checkpoint proc_lsst.multi INFO: cancelling 16896586 on provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:146 [INFO] found job 16896587 in provider checkpoint proc_lsst.multi INFO: found job 16896587 in provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:201 [INFO] cancelling 16896587 on provider checkpoint proc_lsst.multi INFO: cancelling 16896587 on provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:146 [INFO] found job 16896588 in provider checkpoint proc_lsst.multi INFO: found job 16896588 in provider checkpoint 2024-03-07 15:15:39 proc_lsst.multi:201 [INFO] cancelling 16896588 on provider checkpoint proc_lsst.multi INFO: cancelling 16896588 on provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:146 [INFO] found job 16896589 in provider checkpoint proc_lsst.multi INFO: found job 16896589 in provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:201 [INFO] cancelling 16896589 on provider checkpoint proc_lsst.multi INFO: cancelling 16896589 on provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:146 [INFO] found job 16896590 in provider checkpoint proc_lsst.multi INFO: found job 16896590 in provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:201 [INFO] cancelling 16896590 on provider checkpoint proc_lsst.multi INFO: cancelling 16896590 on provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:146 [INFO] found job 16896591 in provider checkpoint proc_lsst.multi INFO: found job 16896591 in provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:201 [INFO] cancelling 16896591 on provider checkpoint proc_lsst.multi INFO: cancelling 16896591 on provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:146 [INFO] found job 16896592 in provider checkpoint proc_lsst.multi INFO: found job 16896592 in provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:201 [INFO] cancelling 16896592 on provider checkpoint proc_lsst.multi INFO: cancelling 16896592 on provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:146 [INFO] found job 16896593 in provider checkpoint proc_lsst.multi INFO: found job 16896593 in provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:201 [INFO] cancelling 16896593 on provider checkpoint proc_lsst.multi INFO: cancelling 16896593 on provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:146 [INFO] found job 16896594 in provider checkpoint proc_lsst.multi INFO: found job 16896594 in provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:201 [INFO] cancelling 16896594 on provider checkpoint proc_lsst.multi INFO: cancelling 16896594 on provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:146 [INFO] found job 16896595 in provider checkpoint proc_lsst.multi INFO: found job 16896595 in provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:201 [INFO] cancelling 16896595 on provider checkpoint proc_lsst.multi INFO: cancelling 16896595 on provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:146 [INFO] found job 16896596 in provider checkpoint proc_lsst.multi INFO: found job 16896596 in provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:201 [INFO] cancelling 16896596 on provider checkpoint proc_lsst.multi INFO: cancelling 16896596 on provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:146 [INFO] found job 16896597 in provider checkpoint proc_lsst.multi INFO: found job 16896597 in provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:201 [INFO] cancelling 16896597 on provider checkpoint proc_lsst.multi INFO: cancelling 16896597 on provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:146 [INFO] found job 16896598 in provider checkpoint proc_lsst.multi INFO: found job 16896598 in provider checkpoint 2024-03-07 15:15:40 proc_lsst.multi:201 [INFO] cancelling 16896598 on provider checkpoint proc_lsst.multi INFO: cancelling 16896598 on provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:146 [INFO] found job 16896599 in provider checkpoint proc_lsst.multi INFO: found job 16896599 in provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:201 [INFO] cancelling 16896599 on provider checkpoint proc_lsst.multi INFO: cancelling 16896599 on provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:146 [INFO] found job 16896600 in provider checkpoint proc_lsst.multi INFO: found job 16896600 in provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:201 [INFO] cancelling 16896600 on provider checkpoint proc_lsst.multi INFO: cancelling 16896600 on provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:146 [INFO] found job 16896601 in provider checkpoint proc_lsst.multi INFO: found job 16896601 in provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:201 [INFO] cancelling 16896601 on provider checkpoint proc_lsst.multi INFO: cancelling 16896601 on provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:146 [INFO] found job 16896602 in provider checkpoint proc_lsst.multi INFO: found job 16896602 in provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:201 [INFO] cancelling 16896602 on provider checkpoint proc_lsst.multi INFO: cancelling 16896602 on provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:146 [INFO] found job 16896603 in provider checkpoint proc_lsst.multi INFO: found job 16896603 in provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:201 [INFO] cancelling 16896603 on provider checkpoint proc_lsst.multi INFO: cancelling 16896603 on provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:146 [INFO] found job 16896604 in provider checkpoint proc_lsst.multi INFO: found job 16896604 in provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:201 [INFO] cancelling 16896604 on provider checkpoint proc_lsst.multi INFO: cancelling 16896604 on provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:146 [INFO] found job 16896605 in provider checkpoint proc_lsst.multi INFO: found job 16896605 in provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:201 [INFO] cancelling 16896605 on provider checkpoint proc_lsst.multi INFO: cancelling 16896605 on provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:146 [INFO] found job 16896606 in provider checkpoint proc_lsst.multi INFO: found job 16896606 in provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:201 [INFO] cancelling 16896606 on provider checkpoint proc_lsst.multi INFO: cancelling 16896606 on provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:146 [INFO] found job 16896607 in provider checkpoint proc_lsst.multi INFO: found job 16896607 in provider checkpoint 2024-03-07 15:15:41 proc_lsst.multi:201 [INFO] cancelling 16896607 on provider checkpoint proc_lsst.multi INFO: cancelling 16896607 on provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:146 [INFO] found job 16896608 in provider checkpoint proc_lsst.multi INFO: found job 16896608 in provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:201 [INFO] cancelling 16896608 on provider checkpoint proc_lsst.multi INFO: cancelling 16896608 on provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:146 [INFO] found job 16896609 in provider checkpoint proc_lsst.multi INFO: found job 16896609 in provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:201 [INFO] cancelling 16896609 on provider checkpoint proc_lsst.multi INFO: cancelling 16896609 on provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:146 [INFO] found job 16896610 in provider checkpoint proc_lsst.multi INFO: found job 16896610 in provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:201 [INFO] cancelling 16896610 on provider checkpoint proc_lsst.multi INFO: cancelling 16896610 on provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:146 [INFO] found job 16896611 in provider checkpoint proc_lsst.multi INFO: found job 16896611 in provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:201 [INFO] cancelling 16896611 on provider checkpoint proc_lsst.multi INFO: cancelling 16896611 on provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:146 [INFO] found job 16896612 in provider checkpoint proc_lsst.multi INFO: found job 16896612 in provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:201 [INFO] cancelling 16896612 on provider checkpoint proc_lsst.multi INFO: cancelling 16896612 on provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:146 [INFO] found job 16896613 in provider checkpoint proc_lsst.multi INFO: found job 16896613 in provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:201 [INFO] cancelling 16896613 on provider checkpoint proc_lsst.multi INFO: cancelling 16896613 on provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:146 [INFO] found job 16896614 in provider checkpoint proc_lsst.multi INFO: found job 16896614 in provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:201 [INFO] cancelling 16896614 on provider checkpoint proc_lsst.multi INFO: cancelling 16896614 on provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:146 [INFO] found job 16896615 in provider checkpoint proc_lsst.multi INFO: found job 16896615 in provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:201 [INFO] cancelling 16896615 on provider checkpoint proc_lsst.multi INFO: cancelling 16896615 on provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:146 [INFO] found job 16896616 in provider checkpoint proc_lsst.multi INFO: found job 16896616 in provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:201 [INFO] cancelling 16896616 on provider checkpoint proc_lsst.multi INFO: cancelling 16896616 on provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:146 [INFO] found job 16896617 in provider checkpoint proc_lsst.multi INFO: found job 16896617 in provider checkpoint 2024-03-07 15:15:42 proc_lsst.multi:201 [INFO] cancelling 16896617 on provider checkpoint proc_lsst.multi INFO: cancelling 16896617 on provider checkpoint 2024-03-07 15:15:43 proc_lsst.multi:146 [INFO] found job 16896618 in provider checkpoint proc_lsst.multi INFO: found job 16896618 in provider checkpoint 2024-03-07 15:15:43 proc_lsst.multi:201 [INFO] cancelling 16896618 on provider checkpoint proc_lsst.multi INFO: cancelling 16896618 on provider checkpoint 2024-03-07 15:15:43 proc_lsst.multi:146 [INFO] found job 16896619 in provider checkpoint proc_lsst.multi INFO: found job 16896619 in provider checkpoint 2024-03-07 15:15:43 proc_lsst.multi:201 [INFO] cancelling 16896619 on provider checkpoint proc_lsst.multi INFO: cancelling 16896619 on provider checkpoint 2024-03-07 15:15:43 proc_lsst.multi:146 [INFO] found job 16896620 in provider checkpoint proc_lsst.multi INFO: found job 16896620 in provider checkpoint 2024-03-07 15:15:43 proc_lsst.multi:201 [INFO] cancelling 16896620 on provider checkpoint proc_lsst.multi INFO: cancelling 16896620 on provider checkpoint 2024-03-07 15:15:43 proc_lsst.multi:146 [INFO] found job 16896621 in provider checkpoint proc_lsst.multi INFO: found job 16896621 in provider checkpoint 2024-03-07 15:15:43 proc_lsst.multi:201 [INFO] cancelling 16896621 on provider checkpoint proc_lsst.multi INFO: cancelling 16896621 on provider checkpoint 2024-03-07 15:15:43 proc_lsst.multi:146 [INFO] found job 16896622 in provider checkpoint proc_lsst.multi INFO: found job 16896622 in provider checkpoint 2024-03-07 15:15:43 proc_lsst.multi:201 [INFO] cancelling 16896622 on provider checkpoint proc_lsst.multi INFO: cancelling 16896622 on provider checkpoint 2024-03-07 15:15:43 proc_lsst.multi:146 [INFO] found job 16896623 in provider checkpoint proc_lsst.multi INFO: found job 16896623 in provider checkpoint 2024-03-07 15:15:43 proc_lsst.multi:201 [INFO] cancelling 16896623 on provider checkpoint proc_lsst.multi INFO: cancelling 16896623 on provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:146 [INFO] found job 16896624 in provider checkpoint proc_lsst.multi INFO: found job 16896624 in provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:201 [INFO] cancelling 16896624 on provider checkpoint proc_lsst.multi INFO: cancelling 16896624 on provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:146 [INFO] found job 16896625 in provider checkpoint proc_lsst.multi INFO: found job 16896625 in provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:201 [INFO] cancelling 16896625 on provider checkpoint proc_lsst.multi INFO: cancelling 16896625 on provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:146 [INFO] found job 16896626 in provider checkpoint proc_lsst.multi INFO: found job 16896626 in provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:201 [INFO] cancelling 16896626 on provider checkpoint proc_lsst.multi INFO: cancelling 16896626 on provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:146 [INFO] found job 16896627 in provider checkpoint proc_lsst.multi INFO: found job 16896627 in provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:201 [INFO] cancelling 16896627 on provider checkpoint proc_lsst.multi INFO: cancelling 16896627 on provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:146 [INFO] found job 16896628 in provider checkpoint proc_lsst.multi INFO: found job 16896628 in provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:201 [INFO] cancelling 16896628 on provider checkpoint proc_lsst.multi INFO: cancelling 16896628 on provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:146 [INFO] found job 16896629 in provider checkpoint proc_lsst.multi INFO: found job 16896629 in provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:201 [INFO] cancelling 16896629 on provider checkpoint proc_lsst.multi INFO: cancelling 16896629 on provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:146 [INFO] found job 16896630 in provider checkpoint proc_lsst.multi INFO: found job 16896630 in provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:201 [INFO] cancelling 16896630 on provider checkpoint proc_lsst.multi INFO: cancelling 16896630 on provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:146 [INFO] found job 16896631 in provider checkpoint proc_lsst.multi INFO: found job 16896631 in provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:201 [INFO] cancelling 16896631 on provider checkpoint proc_lsst.multi INFO: cancelling 16896631 on provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:146 [INFO] found job 16896632 in provider checkpoint proc_lsst.multi INFO: found job 16896632 in provider checkpoint 2024-03-07 15:15:44 proc_lsst.multi:201 [INFO] cancelling 16896632 on provider checkpoint proc_lsst.multi INFO: cancelling 16896632 on provider checkpoint 2024-03-07 15:15:45 proc_lsst.multi:146 [INFO] found job 16896633 in provider checkpoint proc_lsst.multi INFO: found job 16896633 in provider checkpoint 2024-03-07 15:15:45 proc_lsst.multi:201 [INFO] cancelling 16896633 on provider checkpoint proc_lsst.multi INFO: cancelling 16896633 on provider checkpoint 2024-03-07 15:15:45 proc_lsst.multi:146 [INFO] found job 16896634 in provider checkpoint proc_lsst.multi INFO: found job 16896634 in provider checkpoint 2024-03-07 15:15:45 proc_lsst.multi:201 [INFO] cancelling 16896634 on provider checkpoint proc_lsst.multi INFO: cancelling 16896634 on provider checkpoint 2024-03-07 15:15:45 proc_lsst.multi:146 [INFO] found job 16896635 in provider checkpoint proc_lsst.multi INFO: found job 16896635 in provider checkpoint 2024-03-07 15:15:45 proc_lsst.multi:201 [INFO] cancelling 16896635 on provider checkpoint proc_lsst.multi INFO: cancelling 16896635 on provider checkpoint 2024-03-07 15:15:45 proc_lsst.multi:146 [INFO] found job 16896636 in provider checkpoint proc_lsst.multi INFO: found job 16896636 in provider checkpoint 2024-03-07 15:15:45 proc_lsst.multi:201 [INFO] cancelling 16896636 on provider checkpoint proc_lsst.multi INFO: cancelling 16896636 on provider checkpoint 2024-03-07 15:15:45 proc_lsst.multi:146 [INFO] found job 16896637 in provider checkpoint proc_lsst.multi INFO: found job 16896637 in provider checkpoint 2024-03-07 15:15:45 proc_lsst.multi:201 [INFO] cancelling 16896637 on provider checkpoint proc_lsst.multi INFO: cancelling 16896637 on provider checkpoint 2024-03-07 15:15:45 proc_lsst.multi:146 [INFO] found job 16896638 in provider checkpoint proc_lsst.multi INFO: found job 16896638 in provider checkpoint 2024-03-07 15:15:45 proc_lsst.multi:201 [INFO] cancelling 16896638 on provider checkpoint proc_lsst.multi INFO: cancelling 16896638 on provider checkpoint 2024-03-07 15:15:45 proc_lsst.multi:146 [INFO] found job 16896639 in provider checkpoint proc_lsst.multi INFO: found job 16896639 in provider checkpoint 2024-03-07 15:15:45 proc_lsst.multi:201 [INFO] cancelling 16896639 on provider checkpoint proc_lsst.multi INFO: cancelling 16896639 on provider checkpoint 2024-03-07 15:15:45 proc_lsst.multi:146 [INFO] found job 16896640 in provider checkpoint proc_lsst.multi INFO: found job 16896640 in provider checkpoint 2024-03-07 15:15:45 proc_lsst.multi:201 [INFO] cancelling 16896640 on provider checkpoint proc_lsst.multi INFO: cancelling 16896640 on provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:146 [INFO] found job 16896641 in provider checkpoint proc_lsst.multi INFO: found job 16896641 in provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:201 [INFO] cancelling 16896641 on provider checkpoint proc_lsst.multi INFO: cancelling 16896641 on provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:146 [INFO] found job 16896642 in provider checkpoint proc_lsst.multi INFO: found job 16896642 in provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:201 [INFO] cancelling 16896642 on provider checkpoint proc_lsst.multi INFO: cancelling 16896642 on provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:146 [INFO] found job 16896643 in provider checkpoint proc_lsst.multi INFO: found job 16896643 in provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:201 [INFO] cancelling 16896643 on provider checkpoint proc_lsst.multi INFO: cancelling 16896643 on provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:146 [INFO] found job 16896644 in provider checkpoint proc_lsst.multi INFO: found job 16896644 in provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:201 [INFO] cancelling 16896644 on provider checkpoint proc_lsst.multi INFO: cancelling 16896644 on provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:146 [INFO] found job 16896645 in provider checkpoint proc_lsst.multi INFO: found job 16896645 in provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:201 [INFO] cancelling 16896645 on provider checkpoint proc_lsst.multi INFO: cancelling 16896645 on provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:146 [INFO] found job 16896646 in provider checkpoint proc_lsst.multi INFO: found job 16896646 in provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:201 [INFO] cancelling 16896646 on provider checkpoint proc_lsst.multi INFO: cancelling 16896646 on provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:146 [INFO] found job 16896647 in provider checkpoint proc_lsst.multi INFO: found job 16896647 in provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:201 [INFO] cancelling 16896647 on provider checkpoint proc_lsst.multi INFO: cancelling 16896647 on provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:146 [INFO] found job 16896648 in provider checkpoint proc_lsst.multi INFO: found job 16896648 in provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:201 [INFO] cancelling 16896648 on provider checkpoint proc_lsst.multi INFO: cancelling 16896648 on provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:146 [INFO] found job 16896649 in provider checkpoint proc_lsst.multi INFO: found job 16896649 in provider checkpoint 2024-03-07 15:15:46 proc_lsst.multi:201 [INFO] cancelling 16896649 on provider checkpoint proc_lsst.multi INFO: cancelling 16896649 on provider checkpoint 2024-03-07 15:15:47 proc_lsst.multi:146 [INFO] found job 16896650 in provider checkpoint proc_lsst.multi INFO: found job 16896650 in provider checkpoint 2024-03-07 15:15:47 proc_lsst.multi:201 [INFO] cancelling 16896650 on provider checkpoint proc_lsst.multi INFO: cancelling 16896650 on provider checkpoint 2024-03-07 15:15:47 proc_lsst.multi:146 [INFO] found job 16896651 in provider checkpoint proc_lsst.multi INFO: found job 16896651 in provider checkpoint 2024-03-07 15:15:47 proc_lsst.multi:201 [INFO] cancelling 16896651 on provider checkpoint proc_lsst.multi INFO: cancelling 16896651 on provider checkpoint 2024-03-07 15:15:47 proc_lsst.multi:146 [INFO] found job 16896652 in provider checkpoint proc_lsst.multi INFO: found job 16896652 in provider checkpoint 2024-03-07 15:15:47 proc_lsst.multi:201 [INFO] cancelling 16896652 on provider checkpoint proc_lsst.multi INFO: cancelling 16896652 on provider checkpoint 2024-03-07 15:15:47 proc_lsst.multi:146 [INFO] found job 16896653 in provider checkpoint proc_lsst.multi INFO: found job 16896653 in provider checkpoint 2024-03-07 15:15:47 proc_lsst.multi:201 [INFO] cancelling 16896653 on provider checkpoint proc_lsst.multi INFO: cancelling 16896653 on provider checkpoint 2024-03-07 15:15:47 proc_lsst.multi:146 [INFO] found job 16896654 in provider checkpoint proc_lsst.multi INFO: found job 16896654 in provider checkpoint 2024-03-07 15:15:47 proc_lsst.multi:201 [INFO] cancelling 16896654 on provider checkpoint proc_lsst.multi INFO: cancelling 16896654 on provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:146 [INFO] found job 16896655 in provider checkpoint proc_lsst.multi INFO: found job 16896655 in provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:201 [INFO] cancelling 16896655 on provider checkpoint proc_lsst.multi INFO: cancelling 16896655 on provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:146 [INFO] found job 16896656 in provider checkpoint proc_lsst.multi INFO: found job 16896656 in provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:201 [INFO] cancelling 16896656 on provider checkpoint proc_lsst.multi INFO: cancelling 16896656 on provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:146 [INFO] found job 16896657 in provider checkpoint proc_lsst.multi INFO: found job 16896657 in provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:201 [INFO] cancelling 16896657 on provider checkpoint proc_lsst.multi INFO: cancelling 16896657 on provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:146 [INFO] found job 16896658 in provider checkpoint proc_lsst.multi INFO: found job 16896658 in provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:201 [INFO] cancelling 16896658 on provider checkpoint proc_lsst.multi INFO: cancelling 16896658 on provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:146 [INFO] found job 16896659 in provider checkpoint proc_lsst.multi INFO: found job 16896659 in provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:201 [INFO] cancelling 16896659 on provider checkpoint proc_lsst.multi INFO: cancelling 16896659 on provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:146 [INFO] found job 16896660 in provider checkpoint proc_lsst.multi INFO: found job 16896660 in provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:201 [INFO] cancelling 16896660 on provider checkpoint proc_lsst.multi INFO: cancelling 16896660 on provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:146 [INFO] found job 16896661 in provider checkpoint proc_lsst.multi INFO: found job 16896661 in provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:201 [INFO] cancelling 16896661 on provider checkpoint proc_lsst.multi INFO: cancelling 16896661 on provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:146 [INFO] found job 16896662 in provider checkpoint proc_lsst.multi INFO: found job 16896662 in provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:201 [INFO] cancelling 16896662 on provider checkpoint proc_lsst.multi INFO: cancelling 16896662 on provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:146 [INFO] found job 16896663 in provider checkpoint proc_lsst.multi INFO: found job 16896663 in provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:201 [INFO] cancelling 16896663 on provider checkpoint proc_lsst.multi INFO: cancelling 16896663 on provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:146 [INFO] found job 16896664 in provider checkpoint proc_lsst.multi INFO: found job 16896664 in provider checkpoint 2024-03-07 15:15:48 proc_lsst.multi:201 [INFO] cancelling 16896664 on provider checkpoint proc_lsst.multi INFO: cancelling 16896664 on provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:146 [INFO] found job 16896665 in provider checkpoint proc_lsst.multi INFO: found job 16896665 in provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:201 [INFO] cancelling 16896665 on provider checkpoint proc_lsst.multi INFO: cancelling 16896665 on provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:146 [INFO] found job 16896666 in provider checkpoint proc_lsst.multi INFO: found job 16896666 in provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:201 [INFO] cancelling 16896666 on provider checkpoint proc_lsst.multi INFO: cancelling 16896666 on provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:146 [INFO] found job 16896667 in provider checkpoint proc_lsst.multi INFO: found job 16896667 in provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:201 [INFO] cancelling 16896667 on provider checkpoint proc_lsst.multi INFO: cancelling 16896667 on provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:146 [INFO] found job 16896668 in provider checkpoint proc_lsst.multi INFO: found job 16896668 in provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:201 [INFO] cancelling 16896668 on provider checkpoint proc_lsst.multi INFO: cancelling 16896668 on provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:146 [INFO] found job 16896669 in provider checkpoint proc_lsst.multi INFO: found job 16896669 in provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:201 [INFO] cancelling 16896669 on provider checkpoint proc_lsst.multi INFO: cancelling 16896669 on provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:146 [INFO] found job 16896670 in provider checkpoint proc_lsst.multi INFO: found job 16896670 in provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:201 [INFO] cancelling 16896670 on provider checkpoint proc_lsst.multi INFO: cancelling 16896670 on provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:146 [INFO] found job 16896671 in provider checkpoint proc_lsst.multi INFO: found job 16896671 in provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:201 [INFO] cancelling 16896671 on provider checkpoint proc_lsst.multi INFO: cancelling 16896671 on provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:146 [INFO] found job 16896672 in provider checkpoint proc_lsst.multi INFO: found job 16896672 in provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:201 [INFO] cancelling 16896672 on provider checkpoint proc_lsst.multi INFO: cancelling 16896672 on provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:146 [INFO] found job 16896673 in provider checkpoint proc_lsst.multi INFO: found job 16896673 in provider checkpoint 2024-03-07 15:15:49 proc_lsst.multi:201 [INFO] cancelling 16896673 on provider checkpoint proc_lsst.multi INFO: cancelling 16896673 on provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:146 [INFO] found job 16896674 in provider checkpoint proc_lsst.multi INFO: found job 16896674 in provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:201 [INFO] cancelling 16896674 on provider checkpoint proc_lsst.multi INFO: cancelling 16896674 on provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:146 [INFO] found job 16896675 in provider checkpoint proc_lsst.multi INFO: found job 16896675 in provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:201 [INFO] cancelling 16896675 on provider checkpoint proc_lsst.multi INFO: cancelling 16896675 on provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:146 [INFO] found job 16896676 in provider checkpoint proc_lsst.multi INFO: found job 16896676 in provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:201 [INFO] cancelling 16896676 on provider checkpoint proc_lsst.multi INFO: cancelling 16896676 on provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:146 [INFO] found job 16896678 in provider checkpoint proc_lsst.multi INFO: found job 16896678 in provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:201 [INFO] cancelling 16896678 on provider checkpoint proc_lsst.multi INFO: cancelling 16896678 on provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:146 [INFO] found job 16896679 in provider checkpoint proc_lsst.multi INFO: found job 16896679 in provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:201 [INFO] cancelling 16896679 on provider checkpoint proc_lsst.multi INFO: cancelling 16896679 on provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:146 [INFO] found job 16896680 in provider checkpoint proc_lsst.multi INFO: found job 16896680 in provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:201 [INFO] cancelling 16896680 on provider checkpoint proc_lsst.multi INFO: cancelling 16896680 on provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:146 [INFO] found job 16896681 in provider checkpoint proc_lsst.multi INFO: found job 16896681 in provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:201 [INFO] cancelling 16896681 on provider checkpoint proc_lsst.multi INFO: cancelling 16896681 on provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:146 [INFO] found job 16896682 in provider checkpoint proc_lsst.multi INFO: found job 16896682 in provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:201 [INFO] cancelling 16896682 on provider checkpoint proc_lsst.multi INFO: cancelling 16896682 on provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:146 [INFO] found job 16896683 in provider checkpoint proc_lsst.multi INFO: found job 16896683 in provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:201 [INFO] cancelling 16896683 on provider checkpoint proc_lsst.multi INFO: cancelling 16896683 on provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:146 [INFO] found job 16896684 in provider checkpoint proc_lsst.multi INFO: found job 16896684 in provider checkpoint 2024-03-07 15:15:50 proc_lsst.multi:201 [INFO] cancelling 16896684 on provider checkpoint proc_lsst.multi INFO: cancelling 16896684 on provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:146 [INFO] found job 16896685 in provider checkpoint proc_lsst.multi INFO: found job 16896685 in provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:201 [INFO] cancelling 16896685 on provider checkpoint proc_lsst.multi INFO: cancelling 16896685 on provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:146 [INFO] found job 16896686 in provider checkpoint proc_lsst.multi INFO: found job 16896686 in provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:201 [INFO] cancelling 16896686 on provider checkpoint proc_lsst.multi INFO: cancelling 16896686 on provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:146 [INFO] found job 16896687 in provider checkpoint proc_lsst.multi INFO: found job 16896687 in provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:201 [INFO] cancelling 16896687 on provider checkpoint proc_lsst.multi INFO: cancelling 16896687 on provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:146 [INFO] found job 16896688 in provider checkpoint proc_lsst.multi INFO: found job 16896688 in provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:201 [INFO] cancelling 16896688 on provider checkpoint proc_lsst.multi INFO: cancelling 16896688 on provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:146 [INFO] found job 16896689 in provider checkpoint proc_lsst.multi INFO: found job 16896689 in provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:201 [INFO] cancelling 16896689 on provider checkpoint proc_lsst.multi INFO: cancelling 16896689 on provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:146 [INFO] found job 16896690 in provider checkpoint proc_lsst.multi INFO: found job 16896690 in provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:201 [INFO] cancelling 16896690 on provider checkpoint proc_lsst.multi INFO: cancelling 16896690 on provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:146 [INFO] found job 16896691 in provider checkpoint proc_lsst.multi INFO: found job 16896691 in provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:201 [INFO] cancelling 16896691 on provider checkpoint proc_lsst.multi INFO: cancelling 16896691 on provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:146 [INFO] found job 16896692 in provider checkpoint proc_lsst.multi INFO: found job 16896692 in provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:201 [INFO] cancelling 16896692 on provider checkpoint proc_lsst.multi INFO: cancelling 16896692 on provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:146 [INFO] found job 16896693 in provider checkpoint proc_lsst.multi INFO: found job 16896693 in provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:201 [INFO] cancelling 16896693 on provider checkpoint proc_lsst.multi INFO: cancelling 16896693 on provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:146 [INFO] found job 16896694 in provider checkpoint proc_lsst.multi INFO: found job 16896694 in provider checkpoint 2024-03-07 15:15:51 proc_lsst.multi:201 [INFO] cancelling 16896694 on provider checkpoint proc_lsst.multi INFO: cancelling 16896694 on provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:146 [INFO] found job 16896695 in provider checkpoint proc_lsst.multi INFO: found job 16896695 in provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:201 [INFO] cancelling 16896695 on provider checkpoint proc_lsst.multi INFO: cancelling 16896695 on provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:146 [INFO] found job 16896696 in provider checkpoint proc_lsst.multi INFO: found job 16896696 in provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:201 [INFO] cancelling 16896696 on provider checkpoint proc_lsst.multi INFO: cancelling 16896696 on provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:146 [INFO] found job 16896697 in provider checkpoint proc_lsst.multi INFO: found job 16896697 in provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:201 [INFO] cancelling 16896697 on provider checkpoint proc_lsst.multi INFO: cancelling 16896697 on provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:146 [INFO] found job 16896698 in provider checkpoint proc_lsst.multi INFO: found job 16896698 in provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:201 [INFO] cancelling 16896698 on provider checkpoint proc_lsst.multi INFO: cancelling 16896698 on provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:146 [INFO] found job 16896699 in provider checkpoint proc_lsst.multi INFO: found job 16896699 in provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:201 [INFO] cancelling 16896699 on provider checkpoint proc_lsst.multi INFO: cancelling 16896699 on provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:146 [INFO] found job 16896700 in provider checkpoint proc_lsst.multi INFO: found job 16896700 in provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:201 [INFO] cancelling 16896700 on provider checkpoint proc_lsst.multi INFO: cancelling 16896700 on provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:146 [INFO] found job 16896701 in provider checkpoint proc_lsst.multi INFO: found job 16896701 in provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:201 [INFO] cancelling 16896701 on provider checkpoint proc_lsst.multi INFO: cancelling 16896701 on provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:146 [INFO] found job 16896702 in provider checkpoint proc_lsst.multi INFO: found job 16896702 in provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:201 [INFO] cancelling 16896702 on provider checkpoint proc_lsst.multi INFO: cancelling 16896702 on provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:146 [INFO] found job 16896703 in provider checkpoint proc_lsst.multi INFO: found job 16896703 in provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:201 [INFO] cancelling 16896703 on provider checkpoint proc_lsst.multi INFO: cancelling 16896703 on provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:146 [INFO] found job 16896704 in provider checkpoint proc_lsst.multi INFO: found job 16896704 in provider checkpoint 2024-03-07 15:15:52 proc_lsst.multi:201 [INFO] cancelling 16896704 on provider checkpoint proc_lsst.multi INFO: cancelling 16896704 on provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:146 [INFO] found job 16896705 in provider checkpoint proc_lsst.multi INFO: found job 16896705 in provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:201 [INFO] cancelling 16896705 on provider checkpoint proc_lsst.multi INFO: cancelling 16896705 on provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:146 [INFO] found job 16896706 in provider checkpoint proc_lsst.multi INFO: found job 16896706 in provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:201 [INFO] cancelling 16896706 on provider checkpoint proc_lsst.multi INFO: cancelling 16896706 on provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:146 [INFO] found job 16896707 in provider checkpoint proc_lsst.multi INFO: found job 16896707 in provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:201 [INFO] cancelling 16896707 on provider checkpoint proc_lsst.multi INFO: cancelling 16896707 on provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:146 [INFO] found job 16896708 in provider checkpoint proc_lsst.multi INFO: found job 16896708 in provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:201 [INFO] cancelling 16896708 on provider checkpoint proc_lsst.multi INFO: cancelling 16896708 on provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:146 [INFO] found job 16896709 in provider checkpoint proc_lsst.multi INFO: found job 16896709 in provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:201 [INFO] cancelling 16896709 on provider checkpoint proc_lsst.multi INFO: cancelling 16896709 on provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:146 [INFO] found job 16896710 in provider checkpoint proc_lsst.multi INFO: found job 16896710 in provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:201 [INFO] cancelling 16896710 on provider checkpoint proc_lsst.multi INFO: cancelling 16896710 on provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:146 [INFO] found job 16896711 in provider checkpoint proc_lsst.multi INFO: found job 16896711 in provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:201 [INFO] cancelling 16896711 on provider checkpoint proc_lsst.multi INFO: cancelling 16896711 on provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:146 [INFO] found job 16896712 in provider checkpoint proc_lsst.multi INFO: found job 16896712 in provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:201 [INFO] cancelling 16896712 on provider checkpoint proc_lsst.multi INFO: cancelling 16896712 on provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:146 [INFO] found job 16896713 in provider checkpoint proc_lsst.multi INFO: found job 16896713 in provider checkpoint 2024-03-07 15:15:53 proc_lsst.multi:201 [INFO] cancelling 16896713 on provider checkpoint proc_lsst.multi INFO: cancelling 16896713 on provider checkpoint 2024-03-07 15:15:54 proc_lsst.multi:146 [INFO] found job 16896714 in provider checkpoint proc_lsst.multi INFO: found job 16896714 in provider checkpoint 2024-03-07 15:15:54 proc_lsst.multi:201 [INFO] cancelling 16896714 on provider checkpoint proc_lsst.multi INFO: cancelling 16896714 on provider checkpoint 2024-03-07 15:15:54 proc_lsst.multi:146 [INFO] found job 16896385 in provider astro proc_lsst.multi INFO: found job 16896385 in provider astro 2024-03-07 15:15:54 proc_lsst.multi:201 [INFO] cancelling 16896385 on provider astro proc_lsst.multi INFO: cancelling 16896385 on provider astro 2024-03-07 15:15:54 proc_lsst.multi:146 [INFO] found job 16896386 in provider astro proc_lsst.multi INFO: found job 16896386 in provider astro 2024-03-07 15:15:54 proc_lsst.multi:201 [INFO] cancelling 16896386 on provider astro proc_lsst.multi INFO: cancelling 16896386 on provider astro 2024-03-07 15:15:54 proc_lsst.multi:146 [INFO] found job 16896387 in provider astro proc_lsst.multi INFO: found job 16896387 in provider astro 2024-03-07 15:15:54 proc_lsst.multi:201 [INFO] cancelling 16896387 on provider astro proc_lsst.multi INFO: cancelling 16896387 on provider astro 2024-03-07 15:15:54 proc_lsst.multi:146 [INFO] found job 16896388 in provider astro proc_lsst.multi INFO: found job 16896388 in provider astro 2024-03-07 15:15:54 proc_lsst.multi:201 [INFO] cancelling 16896388 on provider astro proc_lsst.multi INFO: cancelling 16896388 on provider astro 2024-03-07 15:15:54 proc_lsst.multi:146 [INFO] found job 16896389 in provider astro proc_lsst.multi INFO: found job 16896389 in provider astro 2024-03-07 15:15:54 proc_lsst.multi:201 [INFO] cancelling 16896389 on provider astro proc_lsst.multi INFO: cancelling 16896389 on provider astro 2024-03-07 15:15:54 proc_lsst.multi:146 [INFO] found job 16896390 in provider astro proc_lsst.multi INFO: found job 16896390 in provider astro 2024-03-07 15:15:54 proc_lsst.multi:201 [INFO] cancelling 16896390 on provider astro proc_lsst.multi INFO: cancelling 16896390 on provider astro 2024-03-07 15:15:54 proc_lsst.multi:146 [INFO] found job 16896392 in provider astro proc_lsst.multi INFO: found job 16896392 in provider astro 2024-03-07 15:15:54 proc_lsst.multi:201 [INFO] cancelling 16896392 on provider astro proc_lsst.multi INFO: cancelling 16896392 on provider astro 2024-03-07 15:15:54 proc_lsst.multi:146 [INFO] found job 16896394 in provider astro proc_lsst.multi INFO: found job 16896394 in provider astro 2024-03-07 15:15:54 proc_lsst.multi:201 [INFO] cancelling 16896394 on provider astro proc_lsst.multi INFO: cancelling 16896394 on provider astro 2024-03-07 15:15:54 proc_lsst.multi:146 [INFO] found job 16896395 in provider astro proc_lsst.multi INFO: found job 16896395 in provider astro 2024-03-07 15:15:54 proc_lsst.multi:201 [INFO] cancelling 16896395 on provider astro proc_lsst.multi INFO: cancelling 16896395 on provider astro 2024-03-07 15:15:55 proc_lsst.multi:146 [INFO] found job 16896396 in provider astro proc_lsst.multi INFO: found job 16896396 in provider astro 2024-03-07 15:15:55 proc_lsst.multi:201 [INFO] cancelling 16896396 on provider astro proc_lsst.multi INFO: cancelling 16896396 on provider astro 2024-03-07 15:15:55 proc_lsst.multi:146 [INFO] found job 16896397 in provider astro proc_lsst.multi INFO: found job 16896397 in provider astro 2024-03-07 15:15:55 proc_lsst.multi:201 [INFO] cancelling 16896397 on provider astro proc_lsst.multi INFO: cancelling 16896397 on provider astro 2024-03-07 15:15:55 proc_lsst.multi:146 [INFO] found job 16896398 in provider astro proc_lsst.multi INFO: found job 16896398 in provider astro 2024-03-07 15:15:55 proc_lsst.multi:201 [INFO] cancelling 16896398 on provider astro proc_lsst.multi INFO: cancelling 16896398 on provider astro 2024-03-07 15:15:55 proc_lsst.multi:146 [INFO] found job 16896399 in provider astro proc_lsst.multi INFO: found job 16896399 in provider astro 2024-03-07 15:15:55 proc_lsst.multi:201 [INFO] cancelling 16896399 on provider astro proc_lsst.multi INFO: cancelling 16896399 on provider astro 2024-03-07 15:15:55 proc_lsst.multi:146 [INFO] found job 16896400 in provider astro proc_lsst.multi INFO: found job 16896400 in provider astro 2024-03-07 15:15:55 proc_lsst.multi:201 [INFO] cancelling 16896400 on provider astro proc_lsst.multi INFO: cancelling 16896400 on provider astro 2024-03-07 15:15:55 proc_lsst.multi:146 [INFO] found job 16896401 in provider astro proc_lsst.multi INFO: found job 16896401 in provider astro 2024-03-07 15:15:55 proc_lsst.multi:201 [INFO] cancelling 16896401 on provider astro proc_lsst.multi INFO: cancelling 16896401 on provider astro 2024-03-07 15:15:55 proc_lsst.multi:146 [INFO] found job 16896402 in provider astro proc_lsst.multi INFO: found job 16896402 in provider astro 2024-03-07 15:15:55 proc_lsst.multi:201 [INFO] cancelling 16896402 on provider astro proc_lsst.multi INFO: cancelling 16896402 on provider astro 2024-03-07 15:15:55 proc_lsst.multi:146 [INFO] found job 16896403 in provider astro proc_lsst.multi INFO: found job 16896403 in provider astro 2024-03-07 15:15:55 proc_lsst.multi:201 [INFO] cancelling 16896403 on provider astro proc_lsst.multi INFO: cancelling 16896403 on provider astro 2024-03-07 15:15:55 proc_lsst.multi:146 [INFO] found job 16896404 in provider astro proc_lsst.multi INFO: found job 16896404 in provider astro 2024-03-07 15:15:55 proc_lsst.multi:201 [INFO] cancelling 16896404 on provider astro proc_lsst.multi INFO: cancelling 16896404 on provider astro 2024-03-07 15:15:55 proc_lsst.multi:146 [INFO] found job 16896405 in provider astro proc_lsst.multi INFO: found job 16896405 in provider astro 2024-03-07 15:15:55 proc_lsst.multi:201 [INFO] cancelling 16896405 on provider astro proc_lsst.multi INFO: cancelling 16896405 on provider astro 2024-03-07 15:15:56 proc_lsst.multi:146 [INFO] found job 16896408 in provider astro proc_lsst.multi INFO: found job 16896408 in provider astro 2024-03-07 15:15:56 proc_lsst.multi:201 [INFO] cancelling 16896408 on provider astro proc_lsst.multi INFO: cancelling 16896408 on provider astro 2024-03-07 15:15:56 proc_lsst.multi:146 [INFO] found job 16896409 in provider astro proc_lsst.multi INFO: found job 16896409 in provider astro 2024-03-07 15:15:56 proc_lsst.multi:201 [INFO] cancelling 16896409 on provider astro proc_lsst.multi INFO: cancelling 16896409 on provider astro 2024-03-07 15:15:56 proc_lsst.multi:146 [INFO] found job 16896410 in provider astro proc_lsst.multi INFO: found job 16896410 in provider astro 2024-03-07 15:15:56 proc_lsst.multi:201 [INFO] cancelling 16896410 on provider astro proc_lsst.multi INFO: cancelling 16896410 on provider astro 2024-03-07 15:15:56 proc_lsst.multi:146 [INFO] found job 16896411 in provider astro proc_lsst.multi INFO: found job 16896411 in provider astro 2024-03-07 15:15:56 proc_lsst.multi:201 [INFO] cancelling 16896411 on provider astro proc_lsst.multi INFO: cancelling 16896411 on provider astro 2024-03-07 15:15:56 proc_lsst.multi:146 [INFO] found job 16896412 in provider astro proc_lsst.multi INFO: found job 16896412 in provider astro 2024-03-07 15:15:56 proc_lsst.multi:201 [INFO] cancelling 16896412 on provider astro proc_lsst.multi INFO: cancelling 16896412 on provider astro 2024-03-07 15:15:56 proc_lsst.multi:146 [INFO] found job 16896413 in provider astro proc_lsst.multi INFO: found job 16896413 in provider astro 2024-03-07 15:15:56 proc_lsst.multi:201 [INFO] cancelling 16896413 on provider astro proc_lsst.multi INFO: cancelling 16896413 on provider astro 2024-03-07 15:15:56 proc_lsst.multi:146 [INFO] found job 16896414 in provider astro proc_lsst.multi INFO: found job 16896414 in provider astro 2024-03-07 15:15:56 proc_lsst.multi:201 [INFO] cancelling 16896414 on provider astro proc_lsst.multi INFO: cancelling 16896414 on provider astro 2024-03-07 15:15:56 proc_lsst.multi:146 [INFO] found job 16896415 in provider astro proc_lsst.multi INFO: found job 16896415 in provider astro 2024-03-07 15:15:56 proc_lsst.multi:201 [INFO] cancelling 16896415 on provider astro proc_lsst.multi INFO: cancelling 16896415 on provider astro 2024-03-07 15:15:57 proc_lsst.multi:146 [INFO] found job 16896416 in provider astro proc_lsst.multi INFO: found job 16896416 in provider astro 2024-03-07 15:15:57 proc_lsst.multi:201 [INFO] cancelling 16896416 on provider astro proc_lsst.multi INFO: cancelling 16896416 on provider astro 2024-03-07 15:15:57 proc_lsst.multi:146 [INFO] found job 16896417 in provider astro proc_lsst.multi INFO: found job 16896417 in provider astro 2024-03-07 15:15:57 proc_lsst.multi:201 [INFO] cancelling 16896417 on provider astro proc_lsst.multi INFO: cancelling 16896417 on provider astro 2024-03-07 15:15:57 proc_lsst.multi:146 [INFO] found job 16896418 in provider astro proc_lsst.multi INFO: found job 16896418 in provider astro 2024-03-07 15:15:57 proc_lsst.multi:201 [INFO] cancelling 16896418 on provider astro proc_lsst.multi INFO: cancelling 16896418 on provider astro 2024-03-07 15:15:57 proc_lsst.multi:146 [INFO] found job 16896424 in provider escience_a40 proc_lsst.multi INFO: found job 16896424 in provider escience_a40 2024-03-07 15:15:57 proc_lsst.multi:201 [INFO] cancelling 16896424 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896424 on provider escience_a40 2024-03-07 15:15:57 proc_lsst.multi:146 [INFO] found job 16896425 in provider escience_a40 proc_lsst.multi INFO: found job 16896425 in provider escience_a40 2024-03-07 15:15:57 proc_lsst.multi:201 [INFO] cancelling 16896425 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896425 on provider escience_a40 2024-03-07 15:15:57 proc_lsst.multi:146 [INFO] found job 16896426 in provider escience_a40 proc_lsst.multi INFO: found job 16896426 in provider escience_a40 2024-03-07 15:15:57 proc_lsst.multi:201 [INFO] cancelling 16896426 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896426 on provider escience_a40 2024-03-07 15:15:57 proc_lsst.multi:146 [INFO] found job 16896427 in provider escience_a40 proc_lsst.multi INFO: found job 16896427 in provider escience_a40 2024-03-07 15:15:57 proc_lsst.multi:201 [INFO] cancelling 16896427 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896427 on provider escience_a40 2024-03-07 15:15:57 proc_lsst.multi:146 [INFO] found job 16896428 in provider escience_a40 proc_lsst.multi INFO: found job 16896428 in provider escience_a40 2024-03-07 15:15:57 proc_lsst.multi:201 [INFO] cancelling 16896428 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896428 on provider escience_a40 2024-03-07 15:15:57 proc_lsst.multi:146 [INFO] found job 16896429 in provider escience_a40 proc_lsst.multi INFO: found job 16896429 in provider escience_a40 2024-03-07 15:15:57 proc_lsst.multi:201 [INFO] cancelling 16896429 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896429 on provider escience_a40 2024-03-07 15:15:57 proc_lsst.multi:146 [INFO] found job 16896430 in provider escience_a40 proc_lsst.multi INFO: found job 16896430 in provider escience_a40 2024-03-07 15:15:57 proc_lsst.multi:201 [INFO] cancelling 16896430 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896430 on provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:146 [INFO] found job 16896431 in provider escience_a40 proc_lsst.multi INFO: found job 16896431 in provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:201 [INFO] cancelling 16896431 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896431 on provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:146 [INFO] found job 16896432 in provider escience_a40 proc_lsst.multi INFO: found job 16896432 in provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:201 [INFO] cancelling 16896432 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896432 on provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:146 [INFO] found job 16896433 in provider escience_a40 proc_lsst.multi INFO: found job 16896433 in provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:201 [INFO] cancelling 16896433 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896433 on provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:146 [INFO] found job 16896434 in provider escience_a40 proc_lsst.multi INFO: found job 16896434 in provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:201 [INFO] cancelling 16896434 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896434 on provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:146 [INFO] found job 16896435 in provider escience_a40 proc_lsst.multi INFO: found job 16896435 in provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:201 [INFO] cancelling 16896435 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896435 on provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:146 [INFO] found job 16896437 in provider escience_a40 proc_lsst.multi INFO: found job 16896437 in provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:201 [INFO] cancelling 16896437 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896437 on provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:146 [INFO] found job 16896438 in provider escience_a40 proc_lsst.multi INFO: found job 16896438 in provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:201 [INFO] cancelling 16896438 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896438 on provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:146 [INFO] found job 16896439 in provider escience_a40 proc_lsst.multi INFO: found job 16896439 in provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:201 [INFO] cancelling 16896439 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896439 on provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:146 [INFO] found job 16896440 in provider escience_a40 proc_lsst.multi INFO: found job 16896440 in provider escience_a40 2024-03-07 15:15:58 proc_lsst.multi:201 [INFO] cancelling 16896440 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896440 on provider escience_a40 2024-03-07 15:15:59 proc_lsst.multi:146 [INFO] found job 16896441 in provider escience_a40 proc_lsst.multi INFO: found job 16896441 in provider escience_a40 2024-03-07 15:15:59 proc_lsst.multi:201 [INFO] cancelling 16896441 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896441 on provider escience_a40 2024-03-07 15:15:59 proc_lsst.multi:146 [INFO] found job 16896442 in provider escience_a40 proc_lsst.multi INFO: found job 16896442 in provider escience_a40 2024-03-07 15:15:59 proc_lsst.multi:201 [INFO] cancelling 16896442 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896442 on provider escience_a40 2024-03-07 15:15:59 proc_lsst.multi:146 [INFO] found job 16896443 in provider escience_a40 proc_lsst.multi INFO: found job 16896443 in provider escience_a40 2024-03-07 15:15:59 proc_lsst.multi:201 [INFO] cancelling 16896443 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896443 on provider escience_a40 2024-03-07 15:15:59 proc_lsst.multi:146 [INFO] found job 16896444 in provider escience_a40 proc_lsst.multi INFO: found job 16896444 in provider escience_a40 2024-03-07 15:15:59 proc_lsst.multi:201 [INFO] cancelling 16896444 on provider escience_a40 proc_lsst.multi INFO: cancelling 16896444 on provider escience_a40 2024-03-07 15:15:59 proc_lsst.multi:146 [INFO] found job 16896419 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896419 in provider escience_rtx6k 2024-03-07 15:15:59 proc_lsst.multi:201 [INFO] cancelling 16896419 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16896419 on provider escience_rtx6k 2024-03-07 15:15:59 proc_lsst.multi:146 [INFO] found job 16896420 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896420 in provider escience_rtx6k 2024-03-07 15:15:59 proc_lsst.multi:201 [INFO] cancelling 16896420 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16896420 on provider escience_rtx6k 2024-03-07 15:15:59 proc_lsst.multi:146 [INFO] found job 16896421 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896421 in provider escience_rtx6k 2024-03-07 15:15:59 proc_lsst.multi:201 [INFO] cancelling 16896421 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16896421 on provider escience_rtx6k 2024-03-07 15:15:59 proc_lsst.multi:146 [INFO] found job 16896422 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896422 in provider escience_rtx6k 2024-03-07 15:15:59 proc_lsst.multi:201 [INFO] cancelling 16896422 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16896422 on provider escience_rtx6k 2024-03-07 15:15:59 proc_lsst.multi:146 [INFO] found job 16896423 in provider escience_rtx6k proc_lsst.multi INFO: found job 16896423 in provider escience_rtx6k 2024-03-07 15:15:59 proc_lsst.multi:201 [INFO] cancelling 16896423 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16896423 on provider escience_rtx6k 2024-03-07 15:15:59 proc_lsst.multi:146 [INFO] found job 53087 in provider local proc_lsst.multi INFO: found job 53087 in provider local 2024-03-07 15:15:59 proc_lsst.multi:201 [INFO] cancelling 53087 on provider local proc_lsst.multi INFO: cancelling 53087 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 53087 parsl.providers.local.local WARNING: Failed to kill PID: 53087 and child processes on local 2024-03-07 15:16:01 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 634.6406 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20220826_B1l_science#step0_20240307T150529Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 634.6478 seconds; current memory usage: 0.339 Gibyte, delta: 0.018 Gibyte, peak delta: 0.018 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 665.9585 seconds; current memory usage: 0.339 Gibyte, delta: 0.203 Gibyte, peak delta: 0.203 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.339 Gibyte (main), 0.339 Gibyte (largest child process) Run Id: None Run Name: DEEP_20220826_B1l_science#step0_20240307T150529Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up